Pyinstaller打包Scrapy專案
阿新 • • 發佈:2019-01-10
1.安裝pyinstaller
2.安裝pywin32
3.安裝其他模組
注意點:
scrapy用pyinstaller打包不能用
cmdline.execute('scrapy crawl douban -o test.csv --nolog'.split())
我用的是CrawlerProcess方式來輸出
舉個栗子:
1、在scrapy專案根目錄下建一個crawl.py(你可以自己定義)如下圖
cralw.py程式碼如下
# -*- coding: utf-8 -*- from scrapy.crawler import CrawlerProcess from scrapy.utils.project import get_project_settings from douban.spiders.douban_spider import Douban_spider #打包需要的import import urllib.robotparser import scrapy.spiderloader import scrapy.statscollectors import scrapy.logformatter import scrapy.dupefilters import scrapy.squeues import scrapy.extensions.spiderstate import scrapy.extensions.corestats import scrapy.extensions.telnet import scrapy.extensions.logstats import scrapy.extensions.memusage import scrapy.extensions.memdebug import scrapy.extensions.feedexport import scrapy.extensions.closespider import scrapy.extensions.debug import scrapy.extensions.httpcache import scrapy.extensions.statsmailer import scrapy.extensions.throttle import scrapy.core.scheduler import scrapy.core.engine import scrapy.core.scraper import scrapy.core.spidermw import scrapy.core.downloader import scrapy.downloadermiddlewares.stats import scrapy.downloadermiddlewares.httpcache import scrapy.downloadermiddlewares.cookies import scrapy.downloadermiddlewares.useragent import scrapy.downloadermiddlewares.httpproxy import scrapy.downloadermiddlewares.ajaxcrawl import scrapy.downloadermiddlewares.chunked import scrapy.downloadermiddlewares.decompression import scrapy.downloadermiddlewares.defaultheaders import scrapy.downloadermiddlewares.downloadtimeout import scrapy.downloadermiddlewares.httpauth import scrapy.downloadermiddlewares.httpcompression import scrapy.downloadermiddlewares.redirect import scrapy.downloadermiddlewares.retry import scrapy.downloadermiddlewares.robotstxt import scrapy.spidermiddlewares.depth import scrapy.spidermiddlewares.httperror import scrapy.spidermiddlewares.offsite import scrapy.spidermiddlewares.referer import scrapy.spidermiddlewares.urllength import scrapy.pipelines import scrapy.core.downloader.handlers.http import scrapy.core.downloader.contextfactory from douban.pipelines import DoubanPipeline from douban.items import DoubanItem import douban.settings if __name__ == '__main__': setting = get_project_settings() process = CrawlerProcess(settings=setting) process.crawl(Douban_spider) process.start()
2、在crawl.py目錄下pyinstaller crawl.py 生成dist,build(可刪)和crawl.spec(可刪)。
3、在crawl.exe目錄下建立資料夾scrapy,然後到自己安裝的scrapy資料夾中把VERSION和mime.types兩個檔案複製到剛才建立的scrapy資料夾中。
4、釋出程式 包括douban/dist 和douban/scrapy.cfg
如果沒有scrapy.cfg無法讀取settings.py和pipelines.py的配置
5、在另外一臺機器上測試成功