requests源碼框架淺析
本文主要是對 requests 的 for human 結構的部分進行簡單分析,對於裏面具體的功能實現(比如cookies如何存儲,http相關對接)沒有深入研究。
1 對於requests主要模塊說明:
1.1 __init__.py: 寫入了requests的各種方法,可以直接調用
1.2 api.py: 定義了requests 的請求的各種方法(get, post, put, delete...)
1.3 session.py: 主要定義了Session類,裏面封裝這各種session的具體請求方法(Session.request()..)
1.4 model.py: 定義了各種模型(Request, Response)
1.5 utils.py: 定義了中途需要用的各種方法
2 requests的具體請求過程
html = requests.get(‘https://www.baidu.com‘, headers=headers).text html2 = requests.Session().get(‘https://www.xxx.com‘, coookies=cookies).text
2.1 requests or session
不管是 requests.get 還是 requests.Session.get 他都會先去 調用調用 session.request(method=method, url=url, **kwargs) 方法
with sessions.Session() as session: return session.request(method=method, url=url, **kwargs)
***此處補一句, 定義方法時候傳入**kwargs這樣類型的參數,很方便後面對象或方法接收參數。***
2.2 session.request 過程
所以現在我們來看 session.request 這個方法
def request(self, method, url, params=None, data=None, headers=None, cookies=None, files=None, auth=None, timeout=None, allow_redirects=True, proxies=None, hooks=None, stream=None, verify=None, cert=None, json=None): """Constructs a :class:`Request <Request>`, prepares it and sends it. Returns :class:`Response <Response>` object. ...""" # Create the Request. req = Request( method = method.upper(), url = url, headers = headers, files = files, data = data or {}, json = json, params = params or {}, auth = auth, cookies = cookies, hooks = hooks, ) prep = self.prepare_request(req) proxies = proxies or {} settings = self.merge_environment_settings( prep.url, proxies, stream, verify, cert ) # Send the request. send_kwargs = { ‘timeout‘: timeout, ‘allow_redirects‘: allow_redirects, } send_kwargs.update(settings) resp = self.send(prep, **send_kwargs) return resp
2.3 Request類
首先將接收到的各種參數(headers, cookies,...)傳入Request類進行封裝
class Request(RequestHooksMixin): def __init__(self, method=None, url=None, headers=None, files=None, data=None, params=None, auth=None, cookies=None, hooks=None, json=None): # Default empty dicts for dict params. data = [] if data is None else data files = [] if files is None else files headers = {} if headers is None else headers params = {} if params is None else params hooks = {} if hooks is None else hooks self.hooks = default_hooks() for (k, v) in list(hooks.items()): self.register_hook(event=k, hook=v) self.method = method self.url = url self.headers = headers self.files = files self.data = data self.json = json self.params = params self.auth = auth self.cookies = cookies # ...
2.4 對request和傳入的參數進行處理
然後將封裝好的對象傳入 prepare_request(self, request) 方法
def prepare_request(self, request): cookies = request.cookies or {} # Bootstrap CookieJar. if not isinstance(cookies, cookielib.CookieJar): cookies = cookiejar_from_dict(cookies) # Merge with session cookies merged_cookies = merge_cookies( merge_cookies(RequestsCookieJar(), self.cookies), cookies) # Set environment‘s basic authentication if not explicitly set. auth = request.auth if self.trust_env and not auth and not self.auth: auth = get_netrc_auth(request.url) p = PreparedRequest() p.prepare( method=request.method.upper(), url=request.url, files=request.files, data=request.data, json=request.json, headers=merge_setting(request.headers, self.headers, dict_class=CaseInsensitiveDict), params=merge_setting(request.params, self.params), auth=merge_setting(auth, self.auth), cookies=merged_cookies, hooks=merge_hooks(request.hooks, self.hooks), ) return p
該方法主要是對傳入個各種參數(headers, cookies,...)進行相應的處理以便後續調用,
這裏他先將cookies進行了處理,然後對所有傳入的參數進行相應處理,調用PrepareRequest.prepare 方法進行處理
def prepare(self, method=None, url=None, headers=None, files=None, data=None, params=None, auth=None, cookies=None, hooks=None, json=None): """Prepares the entire request with the given parameters.""" self.prepare_method(method) self.prepare_url(url, params) self.prepare_headers(headers) self.prepare_cookies(cookies) self.prepare_body(data, files, json) self.prepare_auth(auth, url) # Note that prepare_auth must be last to enable authentication schemes # such as OAuth to work on a fully prepared request. # This MUST go after prepare_auth. Authenticators could add a hook self.prepare_hooks(hooks)
可以看出,每個具體的請求參數都有對應的參數進行處理。
2.5 發送得到response
request請求對象設計好後,send(request),得到 response
requests源碼框架淺析