java httpclient釋放_總結httpclient資源釋放和連線複用
https://blog.csdn.net/weixin_39528029/article/details/114124727
最近修改同事程式碼時遇到一個問題，通過 httpclient 預設配置產生的 httpclient 如果不關閉，會導致連線無法釋放，很快打滿伺服器連線(內嵌 Jetty 配置了 25 連線上限)，主動關閉問題解決；後來優化為通過連線池生成 httpclient 後，如果關閉 httpclient 又會導致連線池關閉，後面新的 httpclient 也無法再請求，這裡總結遇到的一些問題和疑問。
官網示例中的以下三個 close 分別釋放了什麼資源，是否可以省略，以及在什麼時機呼叫，使用連線池時有區別麼？
作為 RPC 通訊客戶端，如何複用 TCP 連線？
一、資源釋放
CloseableHttpClient httpclient = HttpClients.createDefault();
HttpGet httpget = new HttpGet("http://localhost/");
CloseableHttpResponse response = httpclient.execute(httpget);
try {undefined
HttpEntity entity = response.getEntity();
if (entity != null) {undefined
InputStream instream = entity.getContent();
try {undefined
// do something useful
} finally {undefined
instream.close();
}
}
} finally {undefined
response.close();
}
// httpclient.close();
首先需要了解預設配置 createDefault 和使用了 custom 連線池(文章最後的 HttpClientUtil)兩種情況的區別，通過原始碼可以看到前者也建立了連線池，最大連線20個，單個 host最大2個，但是區別在於每次建立的 httpclient 都自己維護了自己的連線池，而 custom 連線池時所有 httpclient 共用同一個連線池，這是在 api 使用方面需要注意的地方，要避免每次請求新建連線池、關閉連線池，造成效能問題。
The difference between closing the content stream and closing the response is that the former will attempt to keep the underlying connection alive by consuming the entity content while the latter immediately shuts down and discards the connection.
第一個 close 是讀取 http 正文的資料流，類似的還有響應寫入流，都需要主動關閉，如果是使用 EntityUtils.toString(response.getEntity(), "UTF-8"); 的方式，其內部會進行關閉。如果還有要讀/寫的資料、或不主動關閉，相當於 http 請求事務未處理完成，這時通過其他方式關閉(第二個 close)相當於異常終止，會導致該連線無法被複用，對比下面兩段日誌。
第一個 close 未呼叫時，第二個 close 呼叫，連線無法被複用，kept alive 0。
o.a.http.impl.execchain.MainClientExec : Connection can be kept alive indefinitely
h.i.c.DefaultManagedHttpClientConnection : http-outgoing-0: Close connection
o.a.http.impl.execchain.MainClientExec : Connection discarded
h.i.c.PoolingHttpClientConnectionManager : Connection released: [id: 0][route: {}->http://127.0.0.1:8080][total kept alive: 0; route allocated: 0 of 2; total allocated: 0 of 20]
第一個 close 正常呼叫時，第二個 close 呼叫，連線可以被複用，kept alive 1。
o.a.http.impl.execchain.MainClientExec : Connection can be kept alive indefinitely
h.i.c.PoolingHttpClientConnectionManager : Connection [id: 0][route: {}->http://127.0.0.1:8080] can be kept alive indefinitely
h.i.c.DefaultManagedHttpClientConnection : http-outgoing-0: set socket timeout to 0
h.i.c.PoolingHttpClientConnectionManager : Connection released: [id: 0][route: {}->http://127.0.0.1:8080][total kept alive: 1; route allocated: 1 of 2; total allocated: 1 of 20]
第二個 close 是強行制止和釋放連線到連線池，相當於對第一個 close 的保底操作(上面關閉了這個似乎沒必要了？)，結合上面引用的官方文件寫到 immediately shuts down and discards the connection，這裡如果判斷需要 keep alive 實際也不會關閉 TCP 連線，因為通過 netstat 可以看到，第二段日誌後在終端可以繼續觀察到連線：
# netstat -n | grep tcp4 | grep 8080
tcp4 0 0 127.0.0.1.8080 127.0.0.1.51003 ESTABLISHED
tcp4 0 0 127.0.0.1.51003 127.0.0.1.8080 ESTABLISHED
在 SOF 上可以搜到這段話，但是感覺和上面觀察到的並不相符？
The underlying HTTP connection is still held by the response object to allow the response content to be streamed directly from the network socket. In order to ensure correct deallocation of system resources, the user MUST call CloseableHttpResponse#close() from a finally clause. Please note that if response content is not fully consumed the underlying connection cannot be safely re-used and will be shut down and discarded by the connection manager.
第三個 clsoe，也就是 httpclient.close 會徹底關閉連線池，以及其中所有連線，一般情況下，只有在關閉應用時呼叫以釋放資源(補充：當 httpClientBuilder.setConnectionManagerShared(true) 時，並不會關閉連線池)。
二、連線複用
根據 http 協議 1.1 版本，各個 web 伺服器都預設支援 keepalive，因此當 http 請求正常完成後，伺服器不會主動關閉 tcp(直到空閒超時或數量達到上限)，使連線會保留一段時間，前面我們也知道 httpclient 在判斷可以 keepalive 後，即使呼叫了 close 也不會關閉 tcp 連線(可以認為 release 到連線池)。為了管理這些保留的連線，以及方便 api 呼叫，一般設定一個全域性的連線池，並基於該連線池提供 httpclient 例項，這樣就不需要考慮維護 httpclient 例項生命週期，隨用隨取(方便狀態管理？)，此外考慮到 http 的單路性，一個請求響應完成結束後，該連線才可以再次複用，因此連線池的最大連線數決定了併發處理量，該配置也是一種保護機制，超出上限的請求會被阻塞，也可以配合熔斷元件使用，當服務方慢、或不健康時熔斷降級。
最後還有一個問題，觀察到 keepalive 的 tcp 連線過一段時間後會變成如下狀態：
# netstat -n | grep tcp4 | grep 8080
tcp4 0 0 127.0.0.1.8080 127.0.0.1.51866 FIN_WAIT_2
tcp4 0 0 127.0.0.1.51866 127.0.0.1.8080 CLOSE_WAIT
可以看出伺服器經過一段時間，認為該連線空閒，因此主動關閉，收到對方響應後進入 FIN_WAIT_2 狀態(等待對方也發起關閉)，而客戶端進入 CLOSE_WAIT 狀態後卻不再發起自己這一方的關閉請求，這時雙方處於半關閉。官方文件解釋如下：
One of the major shortcomings of the classic blocking I/O model is that the network socket can react to I/O events only when blocked in an I/O operation. When a connection is released back to the manager, it can be kept alive however it is unable to monitor the status of the socket and react to any I/O events. If the connection gets closed on the server side, the client side connection is unable to detect the change in the connection state (and react appropriately by closing the socket on its end).
這需要有定期主動做一些檢測和關閉動作，從這個角度考慮，預設配置產生的 HttpClient 沒有這一功能，不應該用於生產環境，下面這個監控執行緒可以完成該工作，包含它的完整的 HttpUtil 從文章最後連接獲取。
public static class IdleConnectionMonitorThread extends Thread {undefined
private final HttpClientConnectionManager connMgr;
private volatile boolean shutdown;
public IdleConnectionMonitorThread(HttpClientConnectionManager connMgr) {undefined
super();
this.connMgr = connMgr;
}
@Override
public void run() {undefined
try {undefined
while (!shutdown) {undefined
synchronized (this) {undefined
wait(30 * 1000);
// Close expired connections
connMgr.closeExpiredConnections();
// Optionally, close connections
// that have been idle longer than 30 sec
connMgr.closeIdleConnections(30, TimeUnit.SECONDS);
}
}
} catch (InterruptedException ex) {undefined
// terminate
}
}
最後展示一個完整的示例，首先多執行緒發起兩個請求，看到建立兩個連線，30秒之後再發起一個請求，可以複用之前其中一個連線，另一個連線因空閒被關閉，隨後最後等待 2 分鐘後再發起一個請求，由於之前連線已過期失效，重新建立連線。
併發兩個請求
16:54:44.504 [ Thread-4] : Connection request: [route: {}->http://127.0.0.1:8080][total kept alive: 0; route allocated: 0 of 150; total allocated: 0 of 150]
16:54:44.504 [ Thread-5] : Connection request: [route: {}->http://127.0.0.1:8080][total kept alive: 0; route allocated: 0 of 150; total allocated: 0 of 150]
16:54:44.515 [ Thread-5] : Connection leased: [id: 1][route: {}->http://127.0.0.1:8080][total kept alive: 0; route allocated: 2 of 150; total allocated: 2 of 150]
16:54:44.515 [ Thread-4] : Connection leased: [id: 0][route: {}->http://127.0.0.1:8080][total kept alive: 0; route allocated: 2 of 150; total allocated: 2 of 150]
16:54:44.517 [ Thread-5] : Opening connection {}->http://127.0.0.1:8080
16:54:44.517 [ Thread-4] : Opening connection {}->http://127.0.0.1:8080
16:54:44.519 [ Thread-4] : Connecting to /127.0.0.1:8080
16:54:44.519 [ Thread-5] : Connecting to /127.0.0.1:8080
16:54:44.521 [ Thread-5] : Connection established 127.0.0.1:52421127.0.0.1:8080
16:54:44.521 [ Thread-4] : Connection established 127.0.0.1:52420127.0.0.1:8080
....
16:54:49.486 [ main] : [leased: 2; pending: 0; available: 0; max: 150]
16:54:49.630 [ Thread-4] : Connection can be kept alive indefinitely
16:54:49.630 [ Thread-5] : Connection can be kept alive indefinitely
16:54:49.633 [ Thread-4] : Connection [id: 0][route: {}->http://127.0.0.1:8080] can be kept alive indefinitely
16:54:49.633 [ Thread-5] : Connection [id: 1][route: {}->http://127.0.0.1:8080] can be kept alive indefinitely
16:54:49.633 [ Thread-4] : http-outgoing-0: set socket timeout to 0
16:54:49.633 [ Thread-5] : http-outgoing-1: set socket timeout to 0
16:54:49.633 [ Thread-4] : Connection released: [id: 0][route: {}->http://127.0.0.1:8080][total kept alive: 1; route allocated: 2 of 150; total allocated: 2 of 150]
16:54:49.633 [ Thread-5] : Connection released: [id: 1][route: {}->http://127.0.0.1:8080][total kept alive: 2; route allocated: 2 of 150; total allocated: 2 of 150]
16:54:54.488 [ main] : [leased: 0; pending: 0; available: 2; max: 150]
#netstat -n | grep tcp4 | grep 8080
tcp4 0 0 127.0.0.1.8080 127.0.0.1.52421 ESTABLISHED
tcp4 0 0 127.0.0.1.8080 127.0.0.1.52420 ESTABLISHED
tcp4 0 0 127.0.0.1.52421 127.0.0.1.8080 ESTABLISHED
tcp4 0 0 127.0.0.1.52420 127.0.0.1.8080 ESTABLISHED
下一個請求
16:55:14.489 [ Thread-6] : Connection request: [route: {}->http://127.0.0.1:8080][total kept alive: 2; route allocated: 2 of 150; total allocated: 2 of 150]
16:55:14.491 [ Thread-6] : http-outgoing-1 << "[read] I/O error: Read timed out"
16:55:14.491 [ Thread-6] : Connection leased: [id: 1][route: {}->http://127.0.0.1:8080][total kept alive: 1; route allocated: 2 of 150; total allocated: 2 of 150]
16:55:14.491 [ Thread-6] : http-outgoing-1: set socket timeout to 0
16:55:14.492 [ Thread-6] : http-outgoing-1: set socket timeout to 8000
.....
16:55:19.501 [ main] : [leased: 1; pending: 0; available: 1; max: 150]
16:55:19.504 [ Thread-6] : Connection can be kept alive indefinitely
16:55:19.504 [ Thread-6] : Connection [id: 1][route: {}->http://127.0.0.1:8080] can be kept alive indefinitely
16:55:19.505 [ Thread-6] : http-outgoing-1: set socket timeout to 0
16:55:19.505 [ Thread-6] : Connection released: [id: 1][route: {}->http://127.0.0.1:8080][total kept alive: 2; route allocated: 2 of 150; total allocated: 2 of 150]
16:55:24.504 [ main] : [leased: 0; pending: 0; available: 2; max: 150]
#netstat -n | grep tcp4 | grep 8080
tcp4 0 0 127.0.0.1.8080 127.0.0.1.52421 ESTABLISHED
tcp4 0 0 127.0.0.1.8080 127.0.0.1.52420 ESTABLISHED
tcp4 0 0 127.0.0.1.52421 127.0.0.1.8080 ESTABLISHED
tcp4 0 0 127.0.0.1.52420 127.0.0.1.8080 ESTABLISHED
複用了上面的連線，下面是隨後逐步超時的日誌。
16:55:39.513 [ main] : [leased: 0; pending: 0; available: 2; max: 150]
16:55:44.491 [ Thread-8] : Closing expired connections
16:55:44.492 [ Thread-8] : Closing connections idle longer than 30 SECONDS
16:55:44.492 [ Thread-8] : http-outgoing-0: Close connection
16:55:44.518 [ main] : [leased: 0; pending: 0; available: 1; max: 150]
....
16:56:09.535 [ main] : [leased: 0; pending: 0; available: 1; max: 150]
16:56:14.499 [ Thread-8] : Closing expired connections
16:56:14.499 [ Thread-8] : Closing connections idle longer than 30 SECONDS
16:56:14.499 [ Thread-8] : http-outgoing-1: Close connection
16:56:14.540 [ main] : [leased: 0; pending: 0; available: 0; max: 150]
分別對應狀態如下，可以看到複用了 52421，隨後 52420 空閒超時被回收，以及最後 52421 也被回收。
#netstat -n | grep tcp4 | grep 8080
tcp4 0 0 127.0.0.1.8080 127.0.0.1.52421 ESTABLISHED
tcp4 0 0 127.0.0.1.52421 127.0.0.1.8080 ESTABLISHED
tcp4 0 0 127.0.0.1.52420 127.0.0.1.8080 TIME_WAIT
...
#netstat -n | grep tcp4 | grep 8080
tcp4 0 0 127.0.0.1.52421 127.0.0.1.8080 TIME_WAIT
最後一個請求後，日誌省略，可以看到是新的連線 52443。
netstat -n | grep tcp4 | grep 8080
tcp4 0 0 127.0.0.1.8080 127.0.0.1.52443 ESTABLISHED
tcp4 0 0 127.0.0.1.52443 127.0.0.1.8080 ESTABLISHED