Introducing Go Language for System Transformation: Tips and Techniques for Improved Performance

Background

At present, most of the interfaces of the background business system work in a synchronous blocking manner, with low resource utilization and limited stand-alone qps. Because the go language natively supports coroutines and can meet both development efficiency and program performance, it was decided to introduce the go language for transformation. Mainly share the following three tips:

1. Encapsulation of C/C++Library

2. Map internal member assignment and protobuf protocol support

3. Network I/O timeout processing

Encapsulation of C/C++Library

Environment construction and syntax are not covered in detail. Read A tour of go to get a good understanding of the go language. The first problem encountered when introducing the go language is how to reuse the existing C/C++basic library that has passed the long-term online inspection. Fortunately, the go language can easily call functions in the C library through Cgo. See Command cgo for specific methods. In the process of use, it is found that Cgo can support C, but cannot perfectly support C++. In the header file included in the import C keyword, if there is a header file containing C++, such as string, Cgo will report an error. In addition, the namespace of C++will also make it impossible for us to access functions or variables under the namespace with the go language. The solution is to wrap C++with C, call C++with C, and call C with go. For example, I have a library libtest. a and test. h written in C++, in which test. h contains strings in C++. At this time, according to the method in Command cgo, introduce the following in the test. go file:

C

Package test

//# cgo LDFLAGS: - L/usr/local/comm/lib - ltest - lstdc++

//# cgo CPPFLAGS: - I/usr/local/comm/include

//# include "test. h"

 

Errors will be reported during compilation. Note that the absolute path is used above, because when we are in the build go project, the relative path is the path when we execute the build command as the starting point. Otherwise, once the directory changes during the build, the Cgo encapsulation will fail. At this time, in order to call the test library, we have to write another t.h and t.cpp (randomly selected)

C

Void C_ Test()// We use this function to call the functions in the test library. The names here are arbitrary

 

In the content of the t.cpp file:

C

#Include "t.h"

#Include "test. h"

Void C_ Test(){

With C_ Test function to encapsulate the function in the test library

}

 

By compiling t.cpp into the target file:

 

C

G++- c t.cpp - Iabcd - Lefg - ltest

 

There are two ways. One is to package t.o into the original libtest. a, and the other is, of course, to package it as an independent static library. The second way is highly recommended here:

 

C

Ar - r libt. a t.o

 

At this time, the call of go to C++library can be completed by introducing t. h and t. a into the corresponding go file.

c

packege test

// #cgo LDFLAGS: -L./ -lt -L/usr/local/comm/lib -ltest -lstdc++

// #cgo CPPFLAGS: -I./ -I/usr/local/comm/include

// #include "t.h"

 

Map internal member assignment and protobuf protocol support

In the process of using the go language, when the value in the map structure is found to be a custom type, the members inside the custom type cannot be "written" (the value returned in the map structure cannot address its members). For example, run the following code:

Package main

Import "fmt"

Type A struct{

T int

}

Func main(){

M:=make (map [int] A)

A:=A {1}

M [1]=a

M [1]. T=2

Fmt. Println (m)

}

 

The compiler will return an error that m [1]. T cannot be assigned. But when the value in the map is a pointer, for example, replace the code in lines 7, 8 and 9 with the following:

M:=make (map [int] * A)

A:=A {1}

M [1]=&a

 

At this time, the assignment of m [1]. T can be completed. In addition, this problem does not exist when the value type in the map structure is a native type of go (that is, byte/int8/16/32/64 float32/64, string, map, slice, etc.), such as:

M:=make (map [int] map [int] int)

Ma:=make (map [int] int)

Ma [1]=1

M [1]=ma

M [1] [1]=-1

Fmt. Println (m)

 

At this time, there is no problem in reading and writing. However, this problem does not exist in slice. In go, after the protobuf protocol file is generated into the corresponding go file, the variables generated for each field are of pointer type. The protobuf protocol has the open source protobuf protocol of Google in git (after all, it's all Google stuff). For details, see the link. Here's a brief talk about the understanding of using pointers. Using pointers can achieve the purpose of throttling to a greater extent. (The benefits of throttling here are faster encoding and decoding speed, faster completion of data packet interaction (because there is less data to be transmitted), and bandwidth saving.), We determine whether the pointer value is nil. If it is nil, we will skip it directly. If it is not nil, it means that we serialize it only when there is data.

Network I/O timeout processing

Due to the complex network environment, we must consider setting timeout processing for packet receiving and packet returning of each request from the client and network requests between internal call chains of the system. Otherwise, the system may have a large number of connections that cannot be released.

1. Timeout setting between http server and client

For an http server, the http package in the go standard library is generally used. It only takes a few lines of code to create an http server, such as:

Http. HandleFunc ("/hello", func (w http. ResponseWriter, r * http. Request){

Io. WriteString (w, "hello world")

});

Http. ListenAndServe()

 

The internal variable DefaultServeMux is used here, but it does not support I/O timeout by default, so we need to create our own http.Server to provide services:

Svr:=&http.Server{

Addr: "0.0.0.0:8080",

ReadTimeout: 4 * time.Second,

WriteTimeout: 4 * time.Second,

}

Svr. ListenAndServe()

 

Wherein, ReadTimeOut is the time from accepting the request to the complete reading of the RequestBody, and WriteTimeOut is the time from the reading of the RequestBody to the complete packet return.

2. Timeout setting of system internal call chain

The internal services of the system often need collaborative services to complete external requests, so communication cannot be avoided. By calling SetDeadline, SetReadDeadline, and SetWriteDeadline to the established connection object net. Connect, you can set the Deadline to timeout each I/O. Once the timeout occurs, you can close the connection object. As the name implies, it is to set the functions of read/write timeout, read timeout and write timeout respectively for the connection object. Their settings are permanent rather than only for one I/O. Once the timeout occurs, a timeout error will be returned. Therefore, they need to be called before each I/O operation. Here is a simple example:

Func SendOnePacket (conn net. Conn, packet [] byte, timeout int64) error{

Conn. SetWriteDeadline (time. Now(). Add (time. Duration (int64 (time. Millisecond) * timeout))

For{

n. Err:=conn.Write (packet)

If err= Nil{

Return errors. New ("Write error:"+err. Error())

}

If n==len (packet){

Return nil

}

Packet=packet [n:]

}

Return nil

}

Func RecvOnePacket (conn net. Conn, timeout int64) ([] byte, error){

Conn. SetReadDeadline (time. Now(). Add (time. Duration (int64 (time. Millisecond) * timeout))

RecvBuff:=bytes. NewBuffer (nil)

Var msgLen int=0

Var buf [4096] byte

For{

n. Err:=conn. Read (buf [0:])

RecvBuf. Write (buf [0: n])

If err= Nil&&err= Io.EOF{

Str:=string (recvBuf. Bytes())

Return nil, errors. New ("Read Error:"+err. Error()+"data:"+str)

}

MsgLen=CheckPacket (recvBuf)

If msgLen>0{

Break

}

If msgLen<0{

Return nil, errors. New ("CheckPacket error, return:"+strconv.FormatInt (int64 (msgLen), 10))

}

}

Packet:=recvBuf. Bytes()

Return packet [: msgLen], nil

}

 

The CheckPacket function is implemented according to its own communication protocol. It is used to check whether the packet is complete. A negative number returned indicates an error. If it is 0, it indicates that it is incomplete. If it is greater than 0, it indicates that the reception is complete, and this value is the full length of the packet. The buf array in RecvOnePacket is very particular. It will determine how many bytes can be read at most every time the system calls the Read function (if the length of the slice passed into the Read function is 0, it will be returned directly). This example does not cache data, so it will avalanche if there is a sticky packet.

Summary

At present, the go language has just been introduced to the team, and the deep understanding of the features of go is still very shallow. In addition, since the first batch of interfaces have just been coded, it is estimated that they will be launched next week, so the specific performance is still unknown.

About Tencent WeTest

Tencent WeTest Quality Open Platform is the official one-stop testing service platform for game developers. We are a dedicated team of experts with more than ten years of experience in quality management. We are committed to the highest quality standards of game development and product quality and tested over 1,000 games.

WeTest integrates cutting-edge tools such as automated testing, compatibility testing, functionality testing, remote device, performance testing, and security testing, covering all testing stages of games throughout their entire life cycle.