Python 爬蟲 urllib模組:get方式
阿新 • • 發佈:2018-12-06
本程式以爬取 百度 首頁為例
格式:
匯入urllib.request
開啟爬取的網頁: response = urllib.request.urlopen('網址')
讀取網頁程式碼: html = response.read()
列印:
1.不decode
print(html) #爬取的網頁程式碼會不分行,沒有空格顯示,很難看
2.decode
print(html.decode()) #爬取的網頁程式碼會分行,像寫規範的程式碼一樣,看起來很舒服
查詢請求結果:
a. response.status # 返回 200:請求成功 404:網頁找不到,請求失敗
b. response.getcode() # 返回 200:請求成功 404:網頁找不到,請求失敗
1.不decode的程式如下:
import urllib.request response = urllib.request.urlopen('www.baidu.com') html = response.read() print(html) print("------------------------------------------------------------------") print("------------------------------------------------------------------") print(response.status)
執行結果:
2.decode的程式如下:
import urllib.request response = urllib.request.urlopen('www.baidu.com') html = response.read() print(html.decode()) print("------------------------------------------------------------------") print("------------------------------------------------------------------") print(response.status)
執行結果:
<!DOCTYPE html> <!--STATUS OK--> <html> <head> <meta http-equiv="content-type" content="text/html;charset=utf-8"> <meta http-equiv="X-UA-Compatible" content="IE=Edge"> <meta content="always" name="referrer"> <meta name="theme-color" content="#2932e1"> <link rel="shortcut icon" href="/favicon.ico" type="image/x-icon" /> <link rel="search" type="application/opensearchdescription+xml" href="/content-search.xml" title="百度搜索" /> <link rel="icon" sizes="any" mask href="//www.baidu.com/img/baidu_85beaf5496f291521eb75ba38eacbd87.svg"> <link rel="dns-prefetch" href="//s1.bdstatic.com"/> <link rel="dns-prefetch" href="//t1.baidu.com"/> <link rel="dns-prefetch" href="//t2.baidu.com"/> <link rel="dns-prefetch" href="//t3.baidu.com"/> <link rel="dns-prefetch" href="//t10.baidu.com"/> <link rel="dns-prefetch" href="//t11.baidu.com"/> <link rel="dns-prefetch" href="//t12.baidu.com"/> <link rel="dns-prefetch" href="//b1.bdstatic.com"/> <title>百度一下,你就知道</title> <style id="css_index" index="index" type="text/css">html,body{height:100%} . . . . </body> </html> ------------------------------------------------------------------ ------------------------------------------------------------------ ------------------------------------------------------------------ 200