問題復現

本地新建request.Session, 多執行緒開啟url併發較高時, 出現以下報錯:

requests.packages.urllib3.connectionpool:Connection pool is full, discarding connection

測試程式碼

def test_pool_max_size():
    # 測試 pool_maxsize 對多執行緒訪問的影響
    def thread_get(url):
        s.get(url)

    s = requests.Session()
    s.mount('https://', HTTPAdapter(pool_connections=1, pool_maxsize=1))
    ts = []
    for _ in range(2):
        t = Thread(target=thread_get, args=('https://www.ask.com',))
        ts.append(t)
        t.start()
    for t in ts:
        t.join()
 

程式碼分析

HTTPAdapter用於對指定網址的連線管理, 引數包含pool_connections=1, pool_maxsize=1.

在HTTPAdapter中的init中會新建poolmanager

    def __init__(self, pool_connections=DEFAULT_POOLSIZE,pool_maxsize=DEFAULT_POOLSIZE ...):
        ...
        self.init_poolmanager(pool_connections, pool_maxsize, block=pool_block)

    def init_poolmanager(self, connections, maxsize ...):
        ...
        self.poolmanager = PoolManager(num_pools=connections, maxsize=maxsize,
                                       block=block, strict=True, **pool_kwargs)
 

在PoolManager中num_pools也就是pool_connections用於構造RecentlyUsedContainer容器, 該容器用於儲存最近使用的HTTPConnectionPool/HTTPSConnectionPool. HTTPConnectionPool/HTTPSConnectionPool管理指定(url, port)的所有連線. 

    def __init__(self, num_pools=10 ...):
        ...
        self.pools = RecentlyUsedContainer(num_pools,
                                           dispose_func=lambda p: p.close())
 

HTTPConnectionPool/HTTPSConnectionPool也有一個pool, 是LifoQueue容器, 由pool_maxsize引數構造, 儲存的是HTTPConnection. HTTPConnection儲存與伺服器的socket連線.

pool_connections, pool_maxsize解析

pool_connections在poolmanager中限制快取中不同url對應的HTTPConnectionPool/HTTPSConnectionPool數目. 

pool_maxsize在HTTPConnectionPool/HTTPSConnectionPool中限制快取中同一個url連線的數目. 單執行緒時執行時, 只會同時存在一個連線, 不會出現連線數過多的問題. 多執行緒時, 當同一網址的url請求數量大於pool_maxsize時, 發起url呼叫時不會報錯, 在請求返回時, 會將連線放入HTTPConnectionPool/HTTPSConnectionPool中的LifoQueue, 當併發請求數量大於pool_maxsize時, LifoQueue不夠放入所有的請求, 就會報錯Connection pool is full, discarding connection.