先了解Scrapy中的Crawler物件體系

Crawler物件

• settings crawler的配置管理器

  • set(name, value, priority=‘project’)
  • setdict(values, priority=‘project’)
  • setmodule(module, priority=‘project’)
  • get(name, default=None)
  • getbool(name, default=False)
  • getint(name, default=0)
  • getfloat(name, default=0.0)
  • getlist(name, default=None)
  • getdict(name, default=None)
  • copy() # 深拷貝當前配置
  • freeze()
  • frozencopy()

• signals crawler的訊號管理器

  • connect(receiver, signal)
  • send_catch_log(signal, **kwargs)
  • send_catch_log_deferred(signal, **kwargs)
  • disconnect(receiver, signal)
  • disconnect_all(signal)

• stats crawler的統計資訊收集器

  • get_value(key, default=None)
  • get_stats()
  • set_value(key, value)
  • set_stats(stats)
  • inc_value(key, count=1, start=0)
  • max_value(key, value)
  • min_value(key, value)
  • clear_stats()
  • open_spider(spider)
  • close_spider(spider)

• extensions 擴充套件管理器，跟蹤所有開啟的擴充套件

• engine 執行引擎，協調crawler的核心邏輯，包括排程，下載和spider

• spider 正在爬取的spider。該spider類的例項由建立crawler時所提供

• crawl(*args, **kwargs) 初始化spider類，啟動執行引擎，啟動crawler

Scrapy內建訊號

engine_started   # 引擎啟動
engine_stopped  # 引擎停止
spider_opened   # spider開始
spider_idle  # spider進入空閒(idle)狀態
spider_closed  # spider被關閉
spider_error  # spider的回撥函式產生錯誤
request_scheduled    # 引擎排程一個 Request
request_dropped   # # 引擎丟棄一個 Request
response_received   # 引擎從downloader獲取到一個新的 Response
response_downloaded  # 當一個 HTTPResponse 被下載
item_scraped   # item通過所有 Item Pipeline 後，沒有被丟棄dropped
item_dropped  #   DropItem丟棄item

以下程式碼實現一個簡單的擴充套件 + 訊號，功能是統計沒有處理的item數量和丟棄的item數量
myextension.py

from scrapy import signals
from scrapy.exceptions import NotConfigured

class SpiderOpenCloseLogging(object):

    def __init__(self):
        self.items_scraped = 0
        self.items_dropped = 0

    @classmethod
    def from_crawler(cls, crawler):
        # 讀取settings配置資訊，檢查是否啟動擴充套件，沒有啟用則丟擲異常，擴充套件被禁用
        if not crawler.settings.getbool('MY_EXTENSION'):
            raise NotConfigured

        # 例項化擴充套件物件
        ext = cls()

        # 註冊訊號處理函式
        crawler.signals.connect(ext.spider_opened, signal=signals.spider_opened)
        crawler.signals.connect(ext.spider_closed, signal=signals.spider_closed)
        crawler.signals.connect(ext.item_scraped, signal=signals.item_scraped)
        crawler.signals.connect(ext.item_dropped, signal=signals.item_dropped)

        return ext

    # 自定義的3個訊號處理函式
    def spider_opened(self, spider):
        spider.log(">>> opened spider %s" % spider.name)

    def spider_closed(self, spider, reason):
        spider.log(">>> closed spider %s" % spider.name)
        spider.log(">>>scraped %d items" % self.items_scraped)
        spider.log(">>>dropped %d items" % self.items_dropped)
        # 獲取狀態收集器資訊
	    print(spider.crawler.stats.get_stats())
	    
    def item_scraped(self, item, response, spider):
        self.items_scraped += 1

    def item_dropped(self, item, response, exception, spider):
        self.items_dropped += 1

啟用擴充套件 settings.py

EXTENSIONS = {
    "scrapys.myextension.SpiderOpenCloseLogging": 100
}

MY_EXTENSION = True

baidu_spider.py

from scrapy import Spider, cmdline
from scrapy_demo.scrapys.items import ScrapysItem


class BaiduSpider(Spider):
    name = "baidu_spider"

    start_urls = [
        "https://www.baidu.com/"
    ]

    def parse(self, response):
        self.crawler.stats.set_value("key", "value")
        return ScrapysItem()


if __name__ == '__main__':
    cmdline.execute("scrapy crawl baidu_spider".split())

item.py中新建item物件

import scrapy

class ScrapysItem(scrapy.Item):
    pass

最後擴充套件中呼叫spider.crawler.stats.get_stats(),可以檢視爬蟲執行後的資訊，打印出的資訊和每次爬蟲執行完的資訊一樣，而且是一個字典物件，可以對這些資料進行獲取

{
'log_count/INFO': 7, 
'start_time': datetime.datetime(2018, 11, 13, 2, 22, 15, 596577), 
'log_count/DEBUG': 6, 
'memusage/startup': 47587328, 
'memusage/max': 47587328, 
'scheduler/enqueued/memory': 1, 
'scheduler/enqueued': 1, 
'scheduler/dequeued/memory': 1, 
'scheduler/dequeued': 1, 
'downloader/request_count': 1, 
'downloader/request_method_count/GET': 1, 
'downloader/request_bytes': 212, 
'downloader/response_count': 1, 
'downloader/response_status_count/200': 1, 
'downloader/response_bytes': 1511, 
'response_received_count': 1, 
'key': 'value', 'log_count/WARNING': 1, 
'item_dropped_count': 1, 
'item_dropped_reasons_count/DropItem': 1, 
'finish_time': datetime.datetime(2018, 11, 13, 2, 22, 16, 113701),
'finish_reason': 'finished'
 }

參考

1. 擴充套件(Extensions)
2. 內建訊號參考手冊(Built-in signals reference)
3. 資料收集(Stats Collection)