1、關於請求url狀態碼重定向問題：

from scrapy import Request

handle_httpstatus_list = [404, 403, 500, 503, 521, 522, 524,301,302]

return Request(self.purl,headers=self.send_headers,meta={‘dont_redirect‘:True}, callback=self.parse)

if response.status in self.handle_httpstatus_list:
    print response.body
    print response.headers[‘Location‘] # 重定向地址
    print response.url # 原始地址

其中 scrapy 自帶 Request 請求遇到302跳轉不會繼續，如果不接收302狀態碼的話，程序將不會執行到parse函數。如果不用settings中的日誌去判斷，你會郁悶死的：
 

LOG_ENABLED = True
LOG_ENCODING = ‘utf-8‘
LOG_FILE = ‘logging.log‘
LOG_LEVEL = ‘DEBUG‘
# LOG_LEVEL = ‘WARNING‘
LOG_STDOUT = False

logging.log:

2017-05-17 17:25:55 [scrapy] INFO: Spider opened
2017-05-17 17:25:55 [scrapy] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2017-05-17 17:25:55 [scrapy] DEBUG: Telnet console listening on 127.0.0.1:6023
2017-05-17 17:26:00 [scrapy] DEBUG: Crawled (302
 
) <GET http://app.58.com/api/list/ershoufang/?tabkey=allcity&action=getListInfo&curVer=7.5.1&isNeedAd=0&ct=filter&os=ios&filterparams=%7B%22param1077%22:%221%22,%22filterLocal%22:%22rongchengqu%22%7D&appId=1&page=1&localname=jy> (referer: None)
2017-05-17 17:26:00 [site58_sale] DEBUG: Read 1 requests from ‘site58_sale‘

scrapy筆記