大資料-爬蟲框架學習--scrapy
阿新 • • 發佈：2019-01-08
--上面就是scrapy最爽的地方，它同時啟動了7個執行緒同時對七篇文章的完整url進行了爬取，這效率想想也是醉了。最關鍵你無需在程式中去指定這些多執行緒的細節，完全有scrapy框架根據實際業務情況自動排程。
{'postdate': [u'2017-04-13 22:13'], 'view': [u'15\u4eba\u9605\u8bfb'], 'link': 'http://blog.csdn.net/firehadoop/article/details/70162692', 'title': [u'\r\n win10\u4e0b\u5b89\u88c5\u914d\u7f6ecisco vpn client \r\n ']}
2017-04-17 20:50:53 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://blog.csdn.net/firehadoop/article/details/69853954> (referer: http://blog.csdn.net/firehadoop?viewmode=contents)
2017-04-17 20:50:53 [scrapy.core.scraper] DEBUG: Scraped from <200 http://blog.csdn.net/firehadoop/article/details/68953541>
{'postdate': [u'2017-04-02 19:22'], 'view': [u'42\u4eba\u9605\u8bfb'], 'link': 'http://blog.csdn.net/firehadoop/article/details/68953541', 'title': [u'\r\n centos7\u4e0b\u8c03\u8bd5\u96c6\u7fa4\u4e09\u53f0\u673a\u5668\u5b9e\u73b0\u514d\u5bc6\u767b\u9646--hadoop\u5b89\u88c5\u7cfb\u5217\u4e4b\u4e00 \r\n ']}
{'postdate': [u'2017-04-09 07:15'], 'view': [u'66\u4eba\u9605\u8bfb'], 'link': 'http://blog.csdn.net/firehadoop/article/details/69791546', 'title': [u'\r\n \u673a\u5668\u5b66\u4e60\u5165\u95e8\u7cfb\u5217-\u4e3a\u4ec0\u4e48\u8981\u673a\u5668\u5b66\u4e60 \r\n ']}
2017-04-17 20:50:53 [scrapy.core.scraper] DEBUG: Scraped from <200 http://blog.csdn.net/firehadoop/article/details/69075792>
{'postdate': [u'2017-04-04 16:19'], 'view': [u'22\u4eba\u9605\u8bfb'], 'link': 'http://blog.csdn.net/firehadoop/article/details/69075792', 'title': [u'\r\n hadoop\u7684dw-hive\u5b89\u88c5\u914d\u7f6e---hadoop\u5b89\u88c5\u7cfb\u5217\u4e4b\u4e09 \r\n ']}
2017-04-17 20:50:53 [scrapy.core.scraper] DEBUG: Scraped from <200 http://blog.csdn.net/firehadoop/article/details/70182483>
{'postdate': [u'2017-04-15 09:11'], 'view': [u'30\u4eba\u9605\u8bfb'], 'link': 'http://blog.csdn.net/firehadoop/article/details/70182483', 'title': [u'\r\n sqoop \u77e5\u8bc6\u53ca\u5b89\u88c5\u914d\u7f6e\u8bf4\u660e--hadoop\u5b89\u88c5\u7cfb\u5217\u4e4b\u56db \r\n ']}
2017-04-17 20:50:53 [scrapy.core.scraper] DEBUG: Scraped from <200 http://blog.csdn.net/firehadoop/article/details/68981887>
{'postdate': [u'2017-04-03 19:31'], 'view': [u'64\u4eba\u9605\u8bfb'], 'link': 'http://blog.csdn.net/firehadoop/article/details/68981887', 'title': [u'\r\n centos7\u5168\u5206\u5e03\u6a21\u5f0f\u5b89\u88c5hadoop--hadoop\u5b89\u88c5\u7cfb\u5217\u4e4b\u4e8c \r\n ']}
2017-04-17 20:50:53 [scrapy.core.scraper] DEBUG: Scraped from <200 http://blog.csdn.net/firehadoop/article/details/69853954>
{'postdate': [u'2017-04-09 17:33'], 'view': [u'18\u4eba\u9605\u8bfb'], 'link': 'http://blog.csdn.net/firehadoop/article/details/69853954', 'title': [u'\r\n \u673a\u5668\u5b66\u4e60\u5165\u95e8\u7cfb\u5217\u4e4b\u4e8c---\u7ebf\u6027\u56de\u5f52\u7b97\u6cd5\u5b66\u4e60 \r\n ']}
2017-04-17 20:50:53 [scrapy.core.engine] INFO: Closing spider (finished)--爬蟲完工了
2017-04-17 20:50:53 [scrapy.extensions.feedexport] INFO:Stored jsonlines feed (7 items) in: csdn5.txt--結果以jsonlines格式儲存在csdn5.txt中，jsonlines就是每行資料一個json，如果才用json格式，會用陣列巢狀的模式把這7行資料放置在一個json中。
2017-04-17 20:50:53 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 2871,
'downloader/request_count': 8,
'downloader/request_method_count/GET': 8,
'downloader/response_bytes': 133247,
'downloader/response_count': 8,
'downloader/response_status_count/200': 8,
'finish_reason': 'finished',
'finish_time': datetime.datetime(2017, 4, 17, 12, 50, 53, 791299),
'item_scraped_count': 7,
'log_count/DEBUG': 16,
'log_count/INFO': 8,
'request_depth_max': 1,
'response_received_count': 8,
'scheduler/dequeued': 8,
'scheduler/dequeued/memory': 8,
'scheduler/enqueued': 8,
'scheduler/enqueued/memory': 8,
'start_time': datetime.datetime(2017, 4, 17, 12, 50, 53, 62819)}
2017-04-17 20:50:53 [scrapy.core.engine] INFO: Spider closed (finished)
{'postdate': [u'2017-04-13 22:13'], 'view': [u'15\u4eba\u9605\u8bfb'], 'link': 'http://blog.csdn.net/firehadoop/article/details/70162692', 'title': [u'\r\n win10\u4e0b\u5b89\u88c5\u914d\u7f6ecisco vpn client \r\n ']}
2017-04-17 20:50:53 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://blog.csdn.net/firehadoop/article/details/69853954> (referer: http://blog.csdn.net/firehadoop?viewmode=contents)
2017-04-17 20:50:53 [scrapy.core.scraper] DEBUG: Scraped from <200 http://blog.csdn.net/firehadoop/article/details/68953541>
{'postdate': [u'2017-04-02 19:22'], 'view': [u'42\u4eba\u9605\u8bfb'], 'link': 'http://blog.csdn.net/firehadoop/article/details/68953541', 'title': [u'\r\n centos7\u4e0b\u8c03\u8bd5\u96c6\u7fa4\u4e09\u53f0\u673a\u5668\u5b9e\u73b0\u514d\u5bc6\u767b\u9646--hadoop\u5b89\u88c5\u7cfb\u5217\u4e4b\u4e00 \r\n ']}
{'postdate': [u'2017-04-09 07:15'], 'view': [u'66\u4eba\u9605\u8bfb'], 'link': 'http://blog.csdn.net/firehadoop/article/details/69791546', 'title': [u'\r\n \u673a\u5668\u5b66\u4e60\u5165\u95e8\u7cfb\u5217-\u4e3a\u4ec0\u4e48\u8981\u673a\u5668\u5b66\u4e60 \r\n ']}
2017-04-17 20:50:53 [scrapy.core.scraper] DEBUG: Scraped from <200 http://blog.csdn.net/firehadoop/article/details/69075792>
{'postdate': [u'2017-04-04 16:19'], 'view': [u'22\u4eba\u9605\u8bfb'], 'link': 'http://blog.csdn.net/firehadoop/article/details/69075792', 'title': [u'\r\n hadoop\u7684dw-hive\u5b89\u88c5\u914d\u7f6e---hadoop\u5b89\u88c5\u7cfb\u5217\u4e4b\u4e09 \r\n ']}
2017-04-17 20:50:53 [scrapy.core.scraper] DEBUG: Scraped from <200 http://blog.csdn.net/firehadoop/article/details/70182483>
{'postdate': [u'2017-04-15 09:11'], 'view': [u'30\u4eba\u9605\u8bfb'], 'link': 'http://blog.csdn.net/firehadoop/article/details/70182483', 'title': [u'\r\n sqoop \u77e5\u8bc6\u53ca\u5b89\u88c5\u914d\u7f6e\u8bf4\u660e--hadoop\u5b89\u88c5\u7cfb\u5217\u4e4b\u56db \r\n ']}
2017-04-17 20:50:53 [scrapy.core.scraper] DEBUG: Scraped from <200 http://blog.csdn.net/firehadoop/article/details/68981887>
{'postdate': [u'2017-04-03 19:31'], 'view': [u'64\u4eba\u9605\u8bfb'], 'link': 'http://blog.csdn.net/firehadoop/article/details/68981887', 'title': [u'\r\n centos7\u5168\u5206\u5e03\u6a21\u5f0f\u5b89\u88c5hadoop--hadoop\u5b89\u88c5\u7cfb\u5217\u4e4b\u4e8c \r\n ']}
2017-04-17 20:50:53 [scrapy.core.scraper] DEBUG: Scraped from <200 http://blog.csdn.net/firehadoop/article/details/69853954>
{'postdate': [u'2017-04-09 17:33'], 'view': [u'18\u4eba\u9605\u8bfb'], 'link': 'http://blog.csdn.net/firehadoop/article/details/69853954', 'title': [u'\r\n \u673a\u5668\u5b66\u4e60\u5165\u95e8\u7cfb\u5217\u4e4b\u4e8c---\u7ebf\u6027\u56de\u5f52\u7b97\u6cd5\u5b66\u4e60 \r\n ']}
2017-04-17 20:50:53 [scrapy.core.engine] INFO: Closing spider (finished)--爬蟲完工了
2017-04-17 20:50:53 [scrapy.extensions.feedexport] INFO:Stored jsonlines feed (7 items) in: csdn5.txt--結果以jsonlines格式儲存在csdn5.txt中，jsonlines就是每行資料一個json，如果才用json格式，會用陣列巢狀的模式把這7行資料放置在一個json中。
2017-04-17 20:50:53 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 2871,
'downloader/request_count': 8,
'downloader/request_method_count/GET': 8,
'downloader/response_bytes': 133247,
'downloader/response_count': 8,
'downloader/response_status_count/200': 8,
'finish_reason': 'finished',
'finish_time': datetime.datetime(2017, 4, 17, 12, 50, 53, 791299),
'item_scraped_count': 7,
'log_count/DEBUG': 16,
'log_count/INFO': 8,
'request_depth_max': 1,
'response_received_count': 8,
'scheduler/dequeued': 8,
'scheduler/dequeued/memory': 8,
'scheduler/enqueued': 8,
'scheduler/enqueued/memory': 8,
'start_time': datetime.datetime(2017, 4, 17, 12, 50, 53, 62819)}
2017-04-17 20:50:53 [scrapy.core.engine] INFO: Spider closed (finished)