2. 程式人生 > >Python爬蟲系列-Requests庫詳解

Python爬蟲系列-Requests庫詳解

阿新 • • 發佈：2018-11-28

Requests基於urllib，比urllib更加方便，可以節約我們大量的工作，完全滿足HTTP測試需求。

例項引入

 import requests
 
 response = requests.get('https://www.baidu.com/')
 print(type(response))
 print(response.status_code)
 print(type(response.text))
 print(response.cookies)

<class 'requests.models.Response'>
200
<class 'str'>
<RequestsCookieJar[<Cookie BDORZ=27315 for .baidu.com/>]>

各種請求方式

 import requests
 requests.post('http://httpbin.org/post')

<Response [200]>

 requests.put('http://httpbin.org/put')

<Response [200]>

 requests.delete('http://httpbin.org/delete')

<Response [200]>

 requests.head('http://httpbin.org/gett')

<Response [404]>

 requests.head('http://httpbin.org/get')

<Response [200]>

 requests.options('http://httpbin.org/get')

<Response [200]>

基本GET請求

 import requests
 
 response = requests.get('http://httpbin.org/get')
 print(response.text)

{
"args": {},
"headers": {
"Accept": "/",
"Accept-Encoding": "gzip, deflate",
"Connection": "close",
"Host": "httpbin.org",
"User-Agent": "python-requests/2.20.1"
},
"origin": "58.34.235.37",
"url": "

http://httpbin.org/get"
}

帶引數的GET請求

import requests
 response = requests.get('http://httpbin.org/get?name=germey&age=22')
 print(response.text)

{
"args": {
"age": "22",
"name": "germey"
},
"headers": {
"Accept": "/",
"Accept-Encoding": "gzip, deflate",
"Connection": "close",
"Host": "httpbin.org",
"User-Agent": "python-requests/2.20.1"
},
"origin": "58.34.235.37",
"url": "http://httpbin.org/get?name=germey&age=22"
}

import requests

 data = { 'name':'germery','age':22 }
 response = requests.get('http://httpbin.org/get',params=data)
 print(response.text)

{
"args": {
"age": "22",
"name": "germery"
},
"headers": {
"Accept": "/",
"Accept-Encoding": "gzip, deflate",
"Connection": "close",
"Host": "httpbin.org",
"User-Agent": "python-requests/2.20.1"
},
"origin": "58.34.235.37",
"url": "http://httpbin.org/get?name=germery&age=22"
}

解析json

 import requests
 
 response = requests.get('http://httpbin.org/get')
 print(type(response.text))

<class 'str'>

 print(response.json())  # 與json.loads(response.text)完全一樣

{'args': {}, 'headers': {'Accept': '/', 'Accept-Encoding': 'gzip, deflate', 'Connection': 'close', 'Host': 'httpbin.org', 'User-Agent': 'python-requests/2.20.1'}, 'origin': '58.34.235.37', 'url': 'http://httpbin.org/get'}

 print(type(response.json()))

<class 'dict'>

獲取二進位制資料

import requests

 response = requests.get('https://github.com/favicon.ico')
 print(type(response.text),type(response.content))

<class 'str'> <class 'bytes'>

下載圖片

import requests
 response = requests.get('https://github.com/favion.ico')
 with open('favicon.ico','wb') as f:
     f.write(response.content)

新增headers

import requests
 headers = { 'User-Agent':'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.77 Safari/537.36' }
 response = requests.get('https://www.zhihu.com/explore',headers=headers)

基本POST請求

import requests
data = { 'name':'germey','age':22 }
response = requests.post('http://httpbin.org/post',data=data)
print(response.text)

{
"args": {},
"data": "",
"files": {},
"form": {
"age": "22",
"name": "germey"
},
"headers": {
"Accept": "/",
"Accept-Encoding": "gzip, deflate",
"Connection": "close",
"Content-Length": "18",
"Content-Type": "application/x-www-form-urlencoded",
"Host": "httpbin.org",
"User-Agent": "python-requests/2.20.1"
},
"json": null,
"origin": "58.34.235.37",
"url": "http://httpbin.org/post"
}

新增headers

 import requests
 
 data = {'name':'germey','age':22}
 headers = { 'User-Agent':'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.77 Safari/537.36' }
 response = requests.post('http://httpbin.org/post',data=data,headers=headers)
 print(response.json())

{'args': {}, 'data': '', 'files': {}, 'form': {'age': '22', 'name': 'germey'}, 'headers': {'Accept': '/', 'Accept-Encoding': 'gzip, deflate', 'Connection': 'close', 'Content-Length': '18', 'Content-Type': 'application/x-www-form-urlencoded', 'Host': 'httpbin.org', 'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.77 Safari/537.36'}, 'json': None, 'origin': '58.34.235.37', 'url': 'http://httpbin.org/post'}

響應

response屬性

import requests

 response = requests.get('http://www.jianshu.com')
print(type(response.status_code),response.status_code)

<class 'int'> 403

 print(type(response.headers),response.headers)

<class 'requests.structures.CaseInsensitiveDict'> {'Date': 'Tue, 27 Nov 2018 20:03:06 GMT', 'Content-Type': 'text/html', 'Transfer-Encoding': 'chunked', 'Connection': 'keep-alive', 'Server': 'Tengine', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains; preload', 'Content-Encoding': 'gzip', 'X-Via': '1.1 dianxinxiazai180:5 (Cdn Cache Server V2.0), 1.1 PSjsntdx3xf38:1 (Cdn Cache Server V2.0)'}

 print(type(response.cookies),response.cookies)

<class 'requests.cookies.RequestsCookieJar'> <RequestsCookieJar[]>

 print(type(response.url),response.url)

<class 'str'> https://www.jianshu.com/

 print(type(response.history),response.history)

<class 'list'> [<Response [301]>]

狀態碼判斷：

 import requests
 
 response = requests.get('http://www.jianshu.com')
 exit() if not response.status_code==requests.codes.ok else print('Request Successfully')
 import requests
 
 response = requests.get('http://www.jianshu.com')
 exit() if not response.status_code==200 else print('Request Successfully')

Requests高階操作

檔案上傳

 import requests
 
 files = {'files':open('favicon.ico','rb')}
 response = requests.post('http://httpbin.org/post',files=files)
 print(response.text)

{
"args": {},
"data": "",
"files": {
"files": "內容省略"
},
"form": {},
"headers": {
"Accept": "/",
"Accept-Encoding": "gzip, deflate",
"Connection": "close",
"Content-Length": "148",
"Content-Type": "multipart/form-data; boundary=6e864227a6fd1cd7a1655802d20d7bd9",
"Host": "httpbin.org",
"User-Agent": "python-requests/2.20.1"
},
"json": null,
"origin": "58.34.235.37",
"url": "http://httpbin.org/post"
}

獲取cookie值

import requests
 
 response = requests.get('http://www.baidu.com')
 print(response.cookies)

<RequestsCookieJar[<Cookie BDORZ=27315 for .baidu.com/>]>

 for key,value in response.cookies.items():
     print(key+'='+value)

BDORZ=27315

會話維持 模擬登入

import requests
 requests.get('http://httpbin.org/cookies/set/number/123456789')

<Response [200]>

 response = requests.get('http://httpbin.org/cookies')
 print(response.text)

{
"cookies": {}
}

s = requests.Session()
 s.get('http://httpbin.org/cookies/set/number/123456789')

<Response [200]>

 response = s.get('http://httpbin.org/cookies')
 print(response.text)

{
"cookies": {
"number": "123456789"
}
}

證書驗證

import requests
 
 response = requests.get('https://www.12306.cn')
 print(response.status_code)
response = requests.get('https://www.12306.cn',verify=False)

/home/dex/.local/lib/python3.6/site-packages/urllib3/connectionpool.py:847: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
InsecureRequestWarning)
/home/dex/.local/lib/python3.6/site-packages/urllib3/connectionpool.py:847: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
InsecureRequestWarning)

新增證書

response = requests.get('https://www.12306.cn',cert=('/path/server.crf','/path/key'))

相關推薦

Python爬蟲系列-Requests庫詳解

Requests基於urllib，比urllib更加方便，可以節約我們大量的工作，完全滿足HTTP測試需求。 例項引入 import requests response = requests.get('https://www.baidu.com/') print(type(response))

Python爬蟲系列-Urllib庫詳解

Urllib庫詳解 Python內建的Http請求庫: * urllib.request 請求模組 * urllib.error 異常處理模組 * urllib.parse url解析模組 * urllib.robotparser robots.txt解析模組 #### 相比在python2基礎上的變化

python爬蟲之requests庫詳解（一，如何通過requests來獲得頁面資訊）

前言： 爬蟲的基礎是與網頁建立聯絡，而我們可以通過get和post兩種方式來建立連線，而我們可以通過引入urllib庫[在python3的環境下匯入的是urllib；而python2的環境下是urllib和urllib2]或者requests庫來實現,從程式的複雜度和可讀性

python爬蟲學習筆記二：Requests庫詳解及HTTP協議

Requests庫的安裝：https://mp.csdn.net/postedit/83715574 r=requests.get(url,params=None,**kwargs) 這個r是Response物件 url ：擬獲取頁面的url連結 params：url中的額外引數

python3 2018分散式爬蟲教程 -4 requests 庫詳解

2.requests 庫 2018分散式爬蟲視訊（崔慶才）+原始碼+電子書下載：https://download.csdn.net/download/qymufeng/10842007 安裝命令： pip install requests requests： 基於 urllib，採⽤

Python3爬蟲學習筆記（2.Requests庫詳解)

Requests庫功能相比Urllib庫更強大，也許是自帶的 如果沒有，cmd輸入pip install requests獲取即可 例項： import requests response = r

【Python爬蟲】Requests庫的安裝

comm AS imp pypi pan span douban OS host 1.按照普通的pip不能行，說find不能 有位小傑控的大牛說了一句：換一個國內的更新源吧， pip install requests -i http://pypi.douban.com/si

Requests庫詳解

OS nbsp 數據 請求 delet 直接 ade imp 比較 urllib庫作為基本庫，requests庫也是在urllib庫基礎上發展的 但是urllib在使用上不如requests便利，比如上篇文章在寫urllib庫的時候，比如代理設置，處理cookie時，沒有寫

Python爬蟲之Requests庫的基本使用

1 import requests 2 response = requests.get('http://www.baidu.com/') 3 print(type(response)) 4 print(response.status_code) 5 print(type(respon

Python爬蟲和情感分析詳解

　　這篇短文的目的是分享我這幾天裡從頭開始學習Python爬蟲技術的經驗，並展示對爬取的文字進行情感分析(文字分類)的一些挖掘結果。 不同於其他專注爬蟲技術的介紹，這裡首先闡述爬取網路資料動機，接著以豆瓣影評為例介紹文字資料的爬取，最後使用文字分類的技術以一種機器學習的方式

[python爬蟲學習] requests庫的使用

前言 昨天寫了一下爬蟲，但發現requests庫的使用有些忘了，所以準備記錄下來！！ 稍微介紹下，給第一次接觸這個庫的人。 您肯定經常看到urllib,urllib2，這些python標準庫。requests庫的功能其實和它們差不多，但方便上手，容易學習！！！

Python 爬蟲基礎Requests庫的使用（二十一）

（一）人性化的Requests庫 在Python爬蟲開發中最為常用的庫就是使用Requests實現HTTP請求，因為Requests實現HTTP請求簡單、操作更為人性化。 （二）get請求的基本用法 def get(url, params=N

Python爬蟲之requests庫(三)：傳送表單資料和JSON資料

import requests 一、傳送表單資料 要傳送表單資料，只需要將一個字典傳遞給引數data payload = {'key1': 'value1', 'key2': 'value

Python爬蟲之requests庫(五)：Cookie、超時、重定向和請求歷史

import requests 一、Cookie 獲取伺服器響應中的cookie資訊 url = 'http://example.com/some/cookie/setting/url'

python爬蟲系列(2.3-requests庫模擬使用者登入)

一、模擬登入拉鉤網 import re import requests class LoginLaGou(object):     """     模擬登入拉鉤網     """    

python爬蟲系列(2.2-requests庫的高階使用)

一、設定代理ip 1、直接在請求的時候加上proxies就可以,注意我們一般會寫上http和https的,這樣當遇到http請求就會走http字典對應的代理 2、具體程式碼 import requests if __name__ == "__main__":  

python爬蟲系列(2.1-requests庫的基本的使用)

一、基本認識 1、傳送一個get請求 import requests if __name__ == "__main__":     # 獲取一個get請求     response = requests.get('http://htt

Python爬蟲系列-BeautifulSoup詳解

安裝 pip3 install beautifulsoup4 解析庫 解析器 使用方法 優勢 劣勢 Python標準庫 BeautifulSoup(markup,'html,parser') Python

Python爬蟲系列-PyQuery詳解

強大又靈活的網頁解析庫。如果你覺得正則寫起來太麻煩，如果你覺得BeautifulSoup語法太難記，如果你熟悉jQuery的語法，那麼PyQuery就是你的最佳選擇。 安裝 pip3 install pyquery 用法講解 字串初始化 html=''' <div> &

python爬蟲urllib庫詳解

什麼是Urllib Urllib是python內建的HTTP請求庫，中文文件如下：https://docs.python.org/3/library/urllib.html包括以下模組urllib.request 請求模組urllib.error 異常處理模組urllib.parse url解析模組urll