入門01-爬取拉勾網頁面的連結
阿新 • • 發佈:2021-06-24
爬蟲的步驟:
1)使用python獲得url的原始碼(向伺服器傳送請求)
2)獲得response的響應物件,獲得響應的原始碼
3)解析原始碼(正則表示式),獲得需要抓取的資料
4)儲存爬取的資源(可寫入到檔案中,也可以寫入到資料庫中)
1 from urllib.request import urlopen 2 import re 3 url = "http://www.lagou.com" 4 response = urlopen(url) 5 # print(response) 6 7 # read() 獲得response物件的原始碼資訊 8 # print(response.read())9 10 # decode將位元組轉換成字串 11 # print(response.read().decode()) 12 13 # 解析原始碼:爬取拉勾網的所有連結 14 html = response.read().decode() 15 16 # r代表字串以原樣輸出,忽略轉義字元 17 # res_url = r"<a.*?href=\".*?\"" 18 res_url = r"<a.*?href=\"(http.*?)\"" 19 20 # re.findall(正則表示式,待匹配字串) 21 urls = re.findall(res_url, html)22 23 # for i in urls: 24 # print(i) 25 # 儲存到txt中 26 # with open("lagou_urls.txt", "wt") as f: 27 # for i in urls: 28 # f.write(i+"\n") 29 30 # 儲存到csv中 31 import csv 32 with open("lagou_urls.csv", "wt", newline="") as f: 33 writer = csv.writer(f) 34 for i in urls: 35 writer.writerow([i])