1. 程式人生 > >爬蟲:python中的requests簡單使用

爬蟲:python中的requests簡單使用

一、requests.get()使用

#Python 的標準庫 urllib 提供了大部分 HTTP 功能,但使用起來較繁瑣。
#通常,我們會使用另外一個優秀的第三方庫:Requests,它的標語是:Requests: HTTP for Humans。
import requests
reponse=requests.get("https://www.baidu.com/")
reponse.encoding='utf-8'
# 返回狀態碼
print(reponse.status_code)
# 列印網頁的內容
print(reponse.text)

二、請求頭

import requests
#構建請求頭,模擬瀏覽器訪問
header={
    'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.162 Safari/537.36'
}
reponse=requests.get('http://www.baidu.com/',headers=header)
print(reponse.content.decode('utf-8'))

三、post請求

import requests
#包裝要傳送的資料
data={
    'name':'zhang',
    'age':'18'
}
header={
    'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.162 Safari/537.36'
}
#呼叫post()方法進行post請求
reponse=requests.post('http://httpbin.org/post',data=data,headers=header)
print(reponse.text)
返回結果如下:傳遞的name和age的值已經被接收
{
  "args": {}, 
  "data": "", 
  "files": {}, 
  "form": {
    "age": "18", 
    "name": "zhang"
  }, 
  "headers": {
    "Accept": "*/*", 
    "Accept-Encoding": "gzip, deflate", 
    "Connection": "close", 
    "Content-Length": "17", 
    "Content-Type": "application/x-www-form-urlencoded", 
    "Host": "httpbin.org", 
    "User-Agent": "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.162 Safari/537.36"
  }, 
  "json": null, 
  "origin": "180.173.37.153", 
  "url": "http://httpbin.org/post"
}

四、代理的使用

import requests
proxy={
    'http':'122.114.31.177'
}
reponse=requests.get('http://www.baidu.com/',proxies=proxy)
print(reponse.status_code)

五、超時設定

from requests.exceptions import Timeout
try:
    reponse=requests.get('http://www.baidu.com/',timeout=0.001)
    print(reponse.status_code)
except Timeout:
    print("訪問超時")