Python爬蟲實現HTTP網路請求多種實現方式

阿新 • • 發佈：2020-06-19

1、通過urllib.requests模組實現傳送請求並讀取網頁內容的簡單示例如下：

#匯入模組
import urllib.request
#開啟需要爬取的網頁
response = urllib.request.urlopen('http://www.baidu.com')
#讀取網頁程式碼
html = response.read()
#列印讀取的內容
print(html)

結果：

b'<!DOCTYPE html><!--STATUS OK-->\n\n\n \n \n       <html><head><meta http-equiv="Content-Type" content="text/html;charset=utf-8"><meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1"><meta content="always" name="referrer"><meta name="theme-color" content="#2932e1"><meta name="description" content="\xe5\x85\xa8\xe7\x90\x83\xe6\x9c\x80\xe5\xa4\xa7\xe7\x9a\x84\xe4\xb8\xad\xe6\x96\x87\xe6\x90\x9c\xe7\xb4\xa2\xe5\xbc\x95\xe6\x93\x8e\xe3\x80\x81\xe8\x87\xb4\xe5\x8a\x9b\xe4\xba\x8e\xe8\xae\xa9\xe7\xbd\x91\xe6\xb0\x91\xe6\x9b\xb4\xe4\xbe\xbf\xe6\x8d\xb7\xe5\x9c\xb0\xe8\x8e\xb7\xe5\x8f\x96\xe4\xbf\xa1\xe6\x81\xaf\xef\xbc\x8c\xe6\x89\xbe\xe5\x88\xb0\xe6\x89\x80\xe6\xb1\x82\xe3\x80\x82\xe7\x99\xbe\xe5\xba\xa6\xe8\xb6\x85\xe8\xbf\x87\xe5\x8d\x83\xe4\xba\xbf\xe7\x9a\x84\xe4\xb8\xad\xe6\x96\x87\xe7\xbd\x91\xe9\xa1\xb5\xe6\x95\xb0\xe6\x8d\xae\xe5\xba\x93\xef\xbc\x8c\xe5\x8f\xaf\xe4\xbb\xa5\xe7\x9e\xac\xe9\x97\xb4\xe6\x89\xbe\xe5\x88\xb0\xe7\x9b\xb8\xe5\x85\xb3\xe7\x9a\x84\xe6\x90\x9c\xe7\xb4\xa2\xe7\xbb\x93\xe6\x9e\x9c\xe3\x80\x82"><link rel="shortcut icon" href="/favicon.ico" rel="external nofollow" type="image/x-icon" /><link rel="search" type="application/opensearchdescription+xml" href="/content-search.xml" rel="external nofollow" title="\xe7\x99\xbe\xe5\xba\xa6\xe6\x90\x9c\xe7\xb4\xa2" /><link rel="icon" sizes="any" mask href="//www.baidu.com/img/baidu_85beaf5496f291521eb75ba38eacbd87.svg" rel="external nofollow" ><link rel="dns-prefetch" href="//dss0.bdstatic.com" rel="external nofollow" /><link rel="dns-prefetch" href="//dss1.bdstatic.com" rel="external nofollow" /><link rel="dns-prefetch" href="//ss1.bdstatic.com" rel="external nofollow" /><link rel="dns-prefetch" href="//sp0.baidu.com" rel="external nofollow" /><link rel="dns-prefetch" href="//sp1.baidu.com" rel="external nofollow" /><link rel="dns-prefetch" href="//sp2.baidu.com" rel="external nofollow" /><title>\xe7\x99\xbe\xe5\xba\xa6\xe4\xb8\x80\xe4\xb8\x8b\xef\xbc\x8c\xe4\xbd\xa0\xe5\xb0\xb1\xe7\x9f\xa5\xe9\x81\x93</title><style index="newi" type="text/css">#form .bdsug{top:39px}.bdsug{display:none;position:absolute;width:535px;background:#fff;border:1px solid 
………………（太多省略）

以上示例中是通過get請求方式獲取百度的網頁內容。

下面是通過urllib.request模組的post請求實現獲取網頁資訊的內容：

#匯入模組
import urllib.parse
import urllib.request
#將資料使用urlencode編碼處理後，再使用encoding設定為utf-8編碼
data = bytes(urllib.parse.urlencode({'word':'hello'}),encoding='utf-8')
#開啟指定需要爬取的網頁
response = urllib.request.urlopen('http://httpbin.org/post',data=data)
html = response.read()
#列印讀取的內容
print(html)

結果：

b'{\n "args": {},\n "data": "",\n "files": {},\n "form": {\n "word": "hello"\n },\n "headers": {\n "Accept-Encoding": "identity",\n "Content-Length": "10",\n "Content-Type": "application/x-www-form-urlencoded",\n "Host": "httpbin.org",\n "User-Agent": "Python-urllib/3.7",\n "X-Amzn-Trace-Id": "Root=1-5ec3f607-00f717e823a5c268fe0e0be8"\n },\n "json": null,\n "origin": "123.139.39.71",\n "url": "http://httpbin.org/post"\n}\n'

2、urllib3模組

通過urllib3模組實現傳送網路請求的示例程式碼：

#匯入模組
import urllib3
#建立PoolManager物件，用於處理與執行緒池的連線以及執行緒安全的所有細節
http = urllib3.PoolManager()
#對需要爬取的網頁傳送請求
response = http.request('GET','https://www.baidu.com/')
#列印讀取的內容
print(response.data)

結果：

b'<!DOCTYPE html><!--STATUS OK-->\r\n<html>\r\n<head>\r\n\t<meta http-equiv="content-type" content="text/html;charset=utf-8">\r\n\t<meta http-equiv="X-UA-Compatible" content="IE=Edge">\r\n\t<link rel="dns-prefetch" href="//s1.bdstatic.com" rel="external nofollow" />\r\n\t<link rel="dns-prefetch" href="//t1.baidu.com" rel="external nofollow" />\r\n\t<link rel="dns-prefetch" href="//t2.baidu.com" rel="external nofollow" />\r\n\t<link rel="dns-prefetch" href="//t3.baidu.com" rel="external nofollow" />\r\n\t<link rel="dns-prefetch" href="//t10.baidu.com" rel="external nofollow" />\r\n\t<link rel="dns-prefetch" href="//t11.baidu.com" rel="external nofollow" />\r\n\t<link rel="dns-prefetch" href="//t12.baidu.com" rel="external nofollow" />\r\n\t<link rel="dns-prefetch" href="//b1.bdstatic.com" rel="external nofollow" />\r\n\t<title>\xe7\x99\xbe\xe5\xba\xa6\xe4\xb8\x80\xe4\xb8\x8b\xef\xbc\x8c\xe4\xbd\xa0\xe5\xb0\xb1\xe7\x9f\xa5\xe9\x81\x93</title>\r\n\t<link href="https://ss1.bdstatic.com/5eN1bjq8AAUYm2zgoY3K/r/www/cache/static/home/css/index.css" rel="external nofollow" rel="stylesheet" type="text/css" />\r\n\t<!--[if lte IE 8]><style index="index" >#content{height:480px\\9}#m{top:260px\\9}</style><![endif]-->\r\n\t<!--[if IE 8]><style index="index" >#u1 a.mnav,#u1 a.mnav:visited{font-family:simsun}</style><![endif]-->\r\n\t<script>var hashMatch = document.location.href.match(/#+(.*wd=[^&].+)/);if (hashMatch && hashMatch[0] && hashMatch[1]) {document.location.replace("http://"+location.host+"/s?"+hashMatch[1]);}
…………………………（太多省略）

post請求實現獲取網頁資訊的內容：

#匯入模組
import urllib3
#建立PoolManager物件，用於處理與執行緒池的連線以及執行緒安全的所有細節
http = urllib3.PoolManager()
#對需要爬取的網頁傳送請求
response = http.request('POST','http://httpbin.org/post',fields={'word':'hello'})
#列印讀取的內容
print(response.data)

結果：

b'{\n "args": {},\n "Content-Length": "128",\n "Content-Type": "multipart/form-data; boundary=06ff68d7a4a22f600244a70bf9382ab2",\n "X-Amzn-Trace-Id": "Root=1-5ec3f8c3-9f33c46c1c1b37f6774b84f2"\n },\n "url": "http://httpbin.org/post"\n}\n'

3、requests模組

以GET請求方式為例，列印多種請求資訊的程式碼:

#匯入模組
import requests
#對需要爬取的網頁傳送請求
response = requests.get('http://www.baidu.com')
#列印狀態碼
print('狀態碼:',response.status_code)
#列印請求url
print('url:',response.url)
#列印頭部資訊
print('header:',response.headers)
#列印cookie資訊
print('cookie:',response.cookies)
#以文字形式列印網頁原始碼
print('text:',response.text)
#以位元組流形式列印網頁原始碼
print('content:',response.content)

結果：

狀態碼: 200
url: http://www.baidu.com/
header: {'Cache-Control': 'private,no-cache,no-store,proxy-revalidate,no-transform','Connection': 'keep-alive','Content-Encoding': 'gzip','Content-Type': 'text/html','Date': 'Tue,19 May 2020 15:28:30 GMT','Last-Modified': 'Mon,23 Jan 2017 13:27:32 GMT','Pragma': 'no-cache','Server': 'bfe/1.0.8.18','Set-Cookie': 'BDORZ=27315; max-age=86400; domain=.baidu.com; path=/','Transfer-Encoding': 'chunked'}
cookie: <RequestsCookieJar[<Cookie BDORZ=27315 for .baidu.com/>]>
text: <!DOCTYPE html>
<!--STATUS OK--><html> <head><meta http-equiv=content-type content=text/html;charset=utf-8>
………………（此處省略）
content: b'<!DOCTYPE html>\r\n<!--STATUS OK--><html> <head><meta http-equiv=content-type content=text/html;charset=utf-8>
………………（此處省略）

以POST請求方式，傳送HTTP網頁請求的示例：

#匯入模組
import requests
#表單引數
data = {'word':'hello'}
#對需要爬取的網頁傳送請求
response = requests.post('http://httpbin.org/post',data=data)
#以位元組流形式列印網頁原始碼
print(response.content)

結果：

b'{\n "args": {},\n "headers": {\n "Accept": "*/*",\n "Accept-Encoding": "gzip,deflate",\n "User-Agent": "python-requests/2.23.0",\n "X-Amzn-Trace-Id": "Root=1-5ec3fc97-965139d919e5a08e8135e731"\n },\n "url": "http://httpbin.org/post"\n}\n'

以上就是本文的全部內容，希望對大家的學習有所幫助，也希望大家多多支援我們。