1. 程式人生 > >requests源碼框架淺析

requests源碼框架淺析

req 淺析 模型 coo 實現 items 請求 ict baidu

本文主要是對 requests 的 for human 結構的部分進行簡單分析,對於裏面具體的功能實現(比如cookies如何存儲,http相關對接)沒有深入研究。

1 對於requests主要模塊說明:

1.1 __init__.py: 寫入了requests的各種方法,可以直接調用

1.2 api.py: 定義了requests 的請求的各種方法(get, post, put, delete...)

1.3 session.py: 主要定義了Session類,裏面封裝這各種session的具體請求方法(Session.request()..)

1.4 model.py: 定義了各種模型(Request, Response)

1.5 utils.py: 定義了中途需要用的各種方法

2 requests的具體請求過程

html = requests.get(https://www.baidu.com, headers=headers).text

html2 = requests.Session().get(https://www.xxx.com, coookies=cookies).text

2.1 requests or session

不管是 requests.get 還是 requests.Session.get 他都會先去 調用調用 session.request(method=method, url=url, **kwargs) 方法

    with sessions.Session() as session:
        return session.request(method=method, url=url, **kwargs)

***此處補一句, 定義方法時候傳入**kwargs這樣類型的參數,很方便後面對象或方法接收參數。***

2.2 session.request 過程

所以現在我們來看 session.request 這個方法

    def request(self, method, url,
        params=None,
        data=None,
        headers
=None, cookies=None, files=None, auth=None, timeout=None, allow_redirects=True, proxies=None, hooks=None, stream=None, verify=None, cert=None, json=None): """Constructs a :class:`Request <Request>`, prepares it and sends it. Returns :class:`Response <Response>` object. ...""" # Create the Request. req = Request( method = method.upper(), url = url, headers = headers, files = files, data = data or {}, json = json, params = params or {}, auth = auth, cookies = cookies, hooks = hooks, ) prep = self.prepare_request(req) proxies = proxies or {} settings = self.merge_environment_settings( prep.url, proxies, stream, verify, cert ) # Send the request. send_kwargs = { timeout: timeout, allow_redirects: allow_redirects, } send_kwargs.update(settings) resp = self.send(prep, **send_kwargs) return resp

2.3 Request類

首先將接收到的各種參數(headers, cookies,...)傳入Request類進行封裝

class Request(RequestHooksMixin):

    def __init__(self, method=None, url=None, headers=None, files=None,
        data=None, params=None, auth=None, cookies=None, hooks=None, json=None):

        # Default empty dicts for dict params.
        data = [] if data is None else data
        files = [] if files is None else files
        headers = {} if headers is None else headers
        params = {} if params is None else params
        hooks = {} if hooks is None else hooks

        self.hooks = default_hooks()
        for (k, v) in list(hooks.items()):
            self.register_hook(event=k, hook=v)

        self.method = method
        self.url = url
        self.headers = headers
        self.files = files
        self.data = data
        self.json = json
        self.params = params
        self.auth = auth
        self.cookies = cookies

    # ...

2.4 對request和傳入的參數進行處理

然後將封裝好的對象傳入 prepare_request(self, request) 方法

    def prepare_request(self, request):

        cookies = request.cookies or {}

        # Bootstrap CookieJar.
        if not isinstance(cookies, cookielib.CookieJar):
            cookies = cookiejar_from_dict(cookies)

        # Merge with session cookies
        merged_cookies = merge_cookies(
            merge_cookies(RequestsCookieJar(), self.cookies), cookies)

        # Set environment‘s basic authentication if not explicitly set.
        auth = request.auth
        if self.trust_env and not auth and not self.auth:
            auth = get_netrc_auth(request.url)

        p = PreparedRequest()
        p.prepare(
            method=request.method.upper(),
            url=request.url,
            files=request.files,
            data=request.data,
            json=request.json,
            headers=merge_setting(request.headers, self.headers, dict_class=CaseInsensitiveDict),
            params=merge_setting(request.params, self.params),
            auth=merge_setting(auth, self.auth),
            cookies=merged_cookies,
            hooks=merge_hooks(request.hooks, self.hooks),
        )
        return p

該方法主要是對傳入個各種參數(headers, cookies,...)進行相應的處理以便後續調用,

這裏他先將cookies進行了處理,然後對所有傳入的參數進行相應處理,調用PrepareRequest.prepare 方法進行處理

    def prepare(self, method=None, url=None, headers=None, files=None,
        data=None, params=None, auth=None, cookies=None, hooks=None, json=None):
        """Prepares the entire request with the given parameters."""

        self.prepare_method(method)
        self.prepare_url(url, params)
        self.prepare_headers(headers)
        self.prepare_cookies(cookies)
        self.prepare_body(data, files, json)
        self.prepare_auth(auth, url)

        # Note that prepare_auth must be last to enable authentication schemes
        # such as OAuth to work on a fully prepared request.

        # This MUST go after prepare_auth. Authenticators could add a hook
        self.prepare_hooks(hooks)

可以看出,每個具體的請求參數都有對應的參數進行處理。

2.5 發送得到response

request請求對象設計好後,send(request),得到 response

requests源碼框架淺析