Python爬蟲-Proxy代理的使用
阿新 • • 發佈:2018-12-08
1 ''' 2 構建代理叢集/佇列 3 每次訪問伺服器,隨機抽取一個代理 4 抽取可以使用 random.choice 5 6 分析步驟: 7 1. 構建代理群 8 2. 每次訪問,隨機選取代理並執行 9 ''' 10 11 12 from urllib import request, error 13 14 15 # 使用代理步驟 16 # 1. 設定代理地址 17 proxy_list = [ 18 # 列表中存放的是dict型別的元素 19 {"http": "101.50.1.2:80"}, 20 {"http": "58.240.172.110:3128"}, 21 {"http": "124.193.51.249:3128"}, 22 {"http": "120.199.64.163:8081"} 23 ] 24 25 # 2. 建立ProxyHandler 26 proxy_handler_list = [] 27 for proxy in proxy_list: 28 proxy_handler = request.ProxyHandler(proxy) 29 proxy_handler_list.append(proxy_handler) 30 # 3. 建立Opener 31 opener_list = []32 for proxy_handler in proxy_handler_list: 33 opener = request.build_opener(proxy_handler) 34 opener_list.append(opener) 35 36 import random 37 38 url = "http://www.baidu.com" 39 # 現在如果訪問url,則使用代理伺服器 40 try: 41 # 4. 安裝Opener 42 opener = random.choice(opener_list) 43 request.install_opener( opener)44 45 rsp = request.urlopen(url) 46 html = rsp.read().decode() 47 print(html) 48 except error.URLError as e: 49 print(e) 50 except Exception as e: 51 print(e)