利用scrapyd管理scrapy的多個爬蟲
阿新 • • 發佈:2019-01-04
說明:環境準備基於 Ubuntu16.04
一、安裝
- sudo pip install scrapyd
- sudo pip install scrapyd-client
二、驗證
命令列輸入:scrapyd
輸出如下表示開啟成功:
bdccl@bdccl-virtual-machine:~$ scrapyd
Removing stale pidfile /home/bdccl/twistd.pid
2017-12-15T19:01:09+0800 [-] Removing stale pidfile /home/bdccl/twistd.pid
2017-12-15T19:01 :09+0800 [-] Loading /usr/local/lib/python2.7/dist-packages/scrapyd/txapp.py...
2017-12-15T19:01:10+0800 [-] Scrapyd web console available at http://127.0.0.1:6800/
2017-12-15T19:01:10+0800 [-] Loaded.
2017-12-15T19:01:10+0800 [twisted.scripts._twistd_unix.UnixAppLogger#info] twistd 17.9.0 (/usr/bin/python 2.7.12) starting up.
2017-12-15T19:01 :10+0800 [twisted.scripts._twistd_unix.UnixAppLogger#info] reactor class: twisted.internet.epollreactor.EPollReactor.
2017-12-15T19:01:10+0800 [-] Site starting on 6800
2017-12-15T19:01:10+0800 [twisted.web.server.Site#info] Starting factory <twisted.web.server.Site instance at 0x7f9589b0fa28>
2017-12-15T19:01:10+0800 [Launcher] Scrapyd 1.2 .0 started: max_proc=4, runner=u'scrapyd.runner'
四、釋出爬蟲
常用命令:
PS:
* 本文只是簡單記錄了一些scrapyd的常用命令,如需深入瞭解scrapyd,建議閱讀scrapyd官方文件*