go net/http Client使用总结
Client数据结构
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60// A Client is an HTTP client. Its zero value (DefaultClient) is a // usable client that uses DefaultTransport. // // The Client's Transport typically has internal state (cached TCP // connections), so Clients should be reused instead of created as // needed. Clients are safe for concurrent use by multiple goroutines. // // A Client is higher-level than a RoundTripper (such as Transport) // and additionally handles HTTP details such as cookies and // redirects. type Client struct { // Transport specifies the mechanism by which individual // HTTP requests are made. // If nil, DefaultTransport is used. Transport RoundTripper // CheckRedirect specifies the policy for handling redirects. // If CheckRedirect is not nil, the client calls it before // following an HTTP redirect. The arguments req and via are // the upcoming request and the requests made already, oldest // first. If CheckRedirect returns an error, the Client's Get // method returns both the previous Response (with its Body // closed) and CheckRedirect's error (wrapped in a url.Error) // instead of issuing the Request req. // As a special case, if CheckRedirect returns ErrUseLastResponse, // then the most recent response is returned with its body // unclosed, along with a nil error. // // If CheckRedirect is nil, the Client uses its default policy, // which is to stop after 10 consecutive requests. CheckRedirect func(req *<span class="" "="">Request, via []*Request) error // Jar specifies the cookie jar. // If Jar is nil, cookies are not sent in requests and ignored // in responses. Jar CookieJar // Timeout specifies a time limit for requests made by this // Client. The timeout includes connection time, any // redirects, and reading the response body. The timer remains // running after Get, Head, Post, or Do return and will // interrupt reading of the Response.Body. // // A Timeout of zero means no timeout. // // The Client cancels requests to the underlying Transport // using the Request.Cancel mechanism. Requests passed ay still set Request.Cancel; both will // cancel the request. // // For compatibility, the Client will also use the deprecated // CancelRequest method on Transport if found. New // RoundTripper implementations should use Request.Cancel // instead of implementing CancelRequest. Timeout time.Duration }
注意Transport
数据结构,对应如下:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136// Transport is an implementation of RoundTripper that supports HTTP, // HTTPS, and HTTP proxies (for either HTTP or HTTPS with CONNECT). // // By default, Transport caches connections for future re-use. // This may leave many open connections when accessing many hosts. // This behavior can be managed using Transport's CloseIdleConnections method // and the MaxIdleConnsPerHost and DisableKeepAlives fields. // // Transports should be reused instead of created as needed. // Transports are safe for concurrent use by multiple goroutines. // // A Transport is a low-level primitive for making HTTP and HTTPS requests. // For high-level functionality, such as cookies and redirects, see Client. // // Transport uses HTTP/1.1 for HTTP URLs and either HTTP/1.1 or HTTP/2 // for HTTPS URLs, depending on whether the server supports HTTP/2. // See the package docs for more about HTTP/2. type Transport struct { idleMu sync.Mutex wantIdle bool // user has requested to close all idle conns idleConn map[connectMethodKey][]*persistConn // most recently used at end idleConnCh map[connectMethodKey]chan *persistConn idleLRU connLRU reqMu sync.Mutex reqCanceler map[*Request]func() altMu sync.RWMutex altProto map[string]RoundTripper // nil or map of URI scheme => RoundTripper // Proxy specifies a function to return a proxy for a given // Request. If the function returns a non-nil error, the // request is aborted with the provided error. // If Proxy is nil or returns a nil *URL, no proxy is used. Proxy func(*Request) (*url.URL, error) // DialContext specifies the dial function for creating unencrypted TCP connections. // If DialContext is nil (and the deprecated Dial below is also nil), // then the transport dials using package net. DialContext func(ctx context.Context, network, addr string) (net.Conn, error) // Dial specifies the dial function for creating unencrypted TCP connections. // // Deprecated: Use DialContext instead, which allows the transport // to cancel dials as soon as they are no longer needed. // If both are set, DialContext takes priority. Dial func(network, addr string) (net.Conn, error) // DialTLS specifies an optional dial function for creating // TLS connections for non-proxied HTTPS requests. // // If DialTLS is nil, Dial and TLSClientConfig are used. // // If DialTLS is set, the Dial hook is not used for HTTPS // requests and the TLSClientConfig and TLSHandshakeTimeout // are ignored. The returned net.Conn is assumed to already be // past the TLS handshake. DialTLS func(network, addr string) (net.Conn, error) // TLSClientConfig specifies the TLS configuration to use with // tls.Client. If nil, the default configuration is used. TLSClientConfig *tls.Config // TLSHandshakeTimeout specifies the maximum amount of time waiting to // wait for a TLS handshake. Zero means no timeout. TLSHandshakeTimeout time.Duration // DisableKeepAlives, if true, prevents re-use of TCP connections // between different HTTP requests. DisableKeepAlives bool // DisableCompression, if true, prevents the Transport from // requesting compression with an "Accept-Encoding: gzip" // request header when the Request contains no existing // Accept-Encoding value. If the Transport requests gzip on // its own and gets a gzipped response, it's transparently // decoded in the Response.Body. However, if the user // explicitly requested gzip it is not automatically // uncompressed. DisableCompression bool // MaxIdleConns controls the maximum number of idle (keep-alive) // connections across all hosts. Zero means no limit. MaxIdleConns int // MaxIdleConnsPerHost, if non-zero, controls the maximum idle // (keep-alive) connections to keep per-host. If zero, // DefaultMaxIdleConnsPerHost is used. MaxIdleConnsPerHost int // IdleConnTimeout is the maximum amount of time an idle // (keep-alive) connection will remain idle before closing // itself. // Zero means no limit. IdleConnTimeout time.Duration // ResponseHeaderTimeout, if non-zero, specifies the amount of // time to wait for a server's response headers after fully // writing the request (including its body, if any). This // time does not include the time to read the response body. ResponseHeaderTimeout time.Duration // ExpectContinueTimeout, if non-zero, specifies the amount of // time to wait for a server's first response headers after fully // writing the request headers if the request has an // "Expect: 100-continue" header. Zero means no timeout. // This time does not include the time to send the request header. ExpectContinueTimeout time.Duration // TLSNextProto specifies how the Transport switches to an // alternate protocol (such as HTTP/2) after a TLS NPN/ALPN // protocol negotiation. If Transport dials an TLS connection // with a non-empty protocol name and TLSNextProto contains a // map entry for that key (such as "h2"), then the func is // called with the request's authority (such as "example.com" // or "example.com:1234") and the TLS connection. The function // must return a RoundTripper that then handles the request. // If TLSNextProto is nil, HTTP/2 support is enabled automatically. TLSNextProto map[string]func(authority string, c *tls.Conn) RoundTripper // MaxResponseHeaderBytes specifies a limit on how many // response bytes are allowed in the server's response // header. // // Zero means to use a default limit. MaxResponseHeaderBytes int64 // nextProtoOnce guards initialization of TLSNextProto and // h2transport (via onceSetNextProtoDefaults) nextProtoOnce sync.Once h2transport *http2Transport // non-nil if http2 wired up // TODO: tunable on max per-host TCP dials in flight (Issue 13957) }
实际上是一个连接池,所以在使用的过程中通常是初始化一个Transport
和Client
,在每个处理函数中都共用该连接池,例如如下Demo:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39package main import ( "bytes" "io/ioutil" "log" "net" "net/http" "time" ) var ( httpClient *http.Client ) // init HTTPClient func init() { httpClient = createHTTPClient() } const ( MaxIdleConns int = 100 MaxIdleConnsPerHost int = 100 IdleConnTimeout int = 90 ) // createHTTPClient for connection re-use func createHTTPClient() *http.Client { client := &http.Client{ Transport: &http.Transport{ Proxy: http.ProxyFromEnvironment, DialContext: (&net.Dialer{ Timeout: 30 * time.Second, KeepAlive: 30 * time.Second, }).DialContext, MaxIdleConns: MaxIdleConns, MaxIdleConnsPerHost: MaxIdleConnsPerHost, IdleConnTimeout: time.Duration(IdleConnTimeout)* time.Second, },
1复制代码Timeout: 20 * time.Second,
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34} return client } func main() { var endPoint string = "https://localhost:8080/doSomething" req, err := http.NewRequest("POST", endPoint, bytes.NewBuffer([]byte("Post this data"))) if err != nil { log.Fatalf("Error Occured. %+v", err) } req.Header.Set("Content-Type", "application/x-www-form-urlencoded") // use httpClient to send request response, err := httpClient.Do(req) if err != nil && response == nil { log.Fatalf("Error sending request to API endpoint. %+v", err) } else { // Close the connection to reuse it defer response.Body.Close() // Let's check if the work actually is done // We have seen inconsistencies even when we get 200 OK response body, err := ioutil.ReadAll(response.Body) if err != nil { log.Fatalf("Couldn't parse response body. %+v", err) } log.Println("Response Body:", string(body)) } }
下面详细讲解Transport
结构需要注意的几个参数设置:
- DisableKeepAlives
1
2
3// DisableKeepAlives, if true, prevents re-use of TCP connections // between different HTTP requests.
表示是否开启http keepalive功能,也即是否重用连接,默认开启(false)
- MaxIdleConns
1
2
3// MaxIdleConns controls the maximum number of idle (keep-alive) // connections across all hosts. Zero means no limit.
表示连接池对所有host的最大链接数量,host也即dest-ip,默认为无穷大(0),但是通常情况下为了性能考虑都要严格限制该数目(实际使用中通常利用压测 二分得到该参数的最佳近似值)。太大容易导致客户端和服务端的socket数量剧增,导致内存吃满,文件描述符不足等问题;太小则限制了连接池的socket数量,资源利用率较低。
- MaxIdleConnsPerHost
1
2
3
4// MaxIdleConnsPerHost, if non-zero, controls the maximum idle // (keep-alive) connections to keep per-host. If zero, // DefaultMaxIdleConnsPerHost is used.
表示连接池对每个host的最大链接数量,从字面意思也可以看出:
1
2MaxIdleConnsPerHost <= MaxIdleConns
如果客户端只需要访问一个host,那么最好将MaxIdleConnsPerHost
与MaxIdleConns
设置为相同,这样逻辑更加清晰。
- IdleConnTimeout
1
2
3
4
5// IdleConnTimeout is the maximum amount of time an idle // (keep-alive) connection will remain idle before closing // itself. // Zero means no limit.
空闲timeout设置,也即socket在该时间内没有交互则自动关闭连接(注意:该timeout起点是从每次空闲开始计时,若有交互则重置为0),该参数通常设置为分钟级别,例如:90秒。
- DialContext
1
2
3
4
5// DialContext specifies the dial function for creating unencrypted TCP connections. // If DialContext is nil (and the deprecated Dial below is also nil), // then the transport dials using package net. DialContext func(ctx context.Context, network, addr string) (net.Conn, error)
该函数用于创建http(非https)连接,通常需要关注Timeout
和KeepAlive
参数。前者表示建立Tcp链接超时时间;后者表示底层为了维持http keepalive状态 每隔多长时间发送Keep-Alive报文。Timeout
通常设置为30s(网络环境良好),KeepAlive
通常设置为30s(与IdleConnTimeout要对应)。
补充说明
Transport
数据结构中的MaxConnsPerHost
参数目前正在研发中(Go项目),其表示对每个host可以发出的最大连接个数(包括长链接和短链接),是非常有用的参数,可以用来对socket数量进行限制,配合MaxIdleConns
和MaxIdleConnsPerHost
一起使用。
net/http Transport MaxConnsPerHost
参考
-
net/http Client使用
-
net/http
-
Transport结构
最后
以上就是激情柚子最近收集整理的关于go net/http Client使用——长连接客户端的使用的全部内容,更多相关go内容请搜索靠谱客的其他文章。
发表评论 取消回复