如何在中間件設置UA池,代理池
阿新 • • 發佈：2019-03-06
請求 rom imp clas quest 代碼 efault windows 返回值
一.UA池：User-Agent池
- 作用：盡可能多的將scrapy工程中的請求偽裝成不同類型的瀏覽器身份。
- 操作流程：
1.在下載中間件中攔截請求
2.將攔截到的請求的請求頭信息中的UA進行篡改偽裝
3.在配置文件中開啟下載中間件
代碼展示：
#導包
from scrapy.contrib.downloadermiddleware.useragent import UserAgentMiddleware
import random
#UA池代碼的編寫（單獨給UA池封裝一個下載中間件的一個類）
class RandomUserAgent(UserAgentMiddleware):
def process_request(self, request, spider):
#從列表中隨機抽選出一個ua值
ua = random.choice(user_agent_list)
#ua值進行當前攔截到請求的ua的寫入操作
request.headers.setdefault(‘User-Agent‘,ua)
user_agent_list = [
"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 "
"(KHTML, like Gecko) Chrome/22.0.1207.1 Safari/537.1",
"Mozilla/5.0 (X11; CrOS i686 2268.111.0) AppleWebKit/536.11 "
"(KHTML, like Gecko) Chrome/20.0.1132.57 Safari/536.11",
"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.6 "
"(KHTML, like Gecko) Chrome/20.0.1092.0 Safari/536.6",
"Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.6 "
"(KHTML, like Gecko) Chrome/20.0.1090.0 Safari/536.6",
"Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.1 "
"(KHTML, like Gecko) Chrome/19.77.34.5 Safari/537.1",
"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/536.5 "
"(KHTML, like Gecko) Chrome/19.0.1084.9 Safari/536.5",
"Mozilla/5.0 (Windows NT 6.0) AppleWebKit/536.5 "
"(KHTML, like Gecko) Chrome/19.0.1084.36 Safari/536.5",
"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.3 "
"(KHTML, like Gecko) Chrome/19.0.1063.0 Safari/536.3",
"Mozilla/5.0 (Windows NT 5.1) AppleWebKit/536.3 "
"(KHTML, like Gecko) Chrome/19.0.1063.0 Safari/536.3",
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_0) AppleWebKit/536.3 "
"(KHTML, like Gecko) Chrome/19.0.1063.0 Safari/536.3",
"Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.3 "
"(KHTML, like Gecko) Chrome/19.0.1062.0 Safari/536.3",
"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.3 "
"(KHTML, like Gecko) Chrome/19.0.1062.0 Safari/536.3",
"Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.3 "
"(KHTML, like Gecko) Chrome/19.0.1061.1 Safari/536.3",
"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.3 "
"(KHTML, like Gecko) Chrome/19.0.1061.1 Safari/536.3",
"Mozilla/5.0 (Windows NT 6.1) AppleWebKit/536.3 "
"(KHTML, like Gecko) Chrome/19.0.1061.1 Safari/536.3",
"Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.3 "
"(KHTML, like Gecko) Chrome/19.0.1061.0 Safari/536.3",
"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/535.24 "
"(KHTML, like Gecko) Chrome/19.0.1055.1 Safari/535.24",
"Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/535.24 "
"(KHTML, like Gecko) Chrome/19.0.1055.1 Safari/535.24"
]
二.代理池
- 作用：盡可能多的將scrapy工程中的請求的IP設置成不同的。
- 操作流程：
1.在下載中間件中攔截請求
2.將攔截到的請求的IP修改成某一代理IP
3.在配置文件中開啟下載中間件
代碼展示：
#批量對攔截到的請求進行ip更換
#單獨封裝下載中間件類
class Proxy(object):
def process_request(self, request, spider):
#對攔截到請求的url進行判斷（協議頭到底是http還是https）
#request.url返回值：http://www.xxx.com
h = request.url.split(‘:‘)[0] #請求的協議頭
if h == ‘https‘:
ip = random.choice(PROXY_https)
request.meta[‘proxy‘] = ‘https://‘+ip
else:
ip = random.choice(PROXY_http)
request.meta[‘proxy‘] = ‘http://‘ + ip
#可被選用的代理IP
PROXY_http = [
‘153.180.102.104:80‘,
‘195.208.131.189:56055‘,
]
PROXY_https = [
‘120.83.49.90:9000‘,
‘95.189.112.214:35508‘,
]
如何在中間件設置UA池,代理池