Scrapy 爬取圖片例項
阿新 • • 發佈:2018-12-07
目標:360攝影美圖
建立scrapy:
scrapy startproject images360
建立spider:
scrapy genspider images images.so.com
修改程式碼:
修改spider:修改images.py:程式碼是根據下拉網頁的AJAX請求分析出來的。
# -*- coding: utf-8 -*- from scrapy import Spider, Request from urllib.parse import urlencode import json from images360.items import ImageItem class ImagesSpider(Spider): name = 'images' allowed_domains = ['images.so.com'] start_urls = ['http://images.so.com/'] def start_requests(self): data = {'ch': 'beauty', 'listtype': 'new'} base_url = 'https://image.so.com/zj?' for page in range(1, self.settings.get('MAX_PAGE') + 1): data['sn'] = page * 30 params = urlencode(data) url = base_url + params yield Request(url, self.parse) def parse(self, response): result = json.loads(response.text) for image in result.get('list'): item = ImageItem() item['id'] = image.get('imageid') item['url'] = image.get('qhimg_url') item['title'] = image.get('group_title') item['thumb'] = image.get('qhimg_thumb_url') yield item
修改items.py:想要得到的欄位
from scrapy import Item,Field
class ImageItem(Item):
collection = table = 'images'
id = Field()
url = Field()
title = Field()
thumb = Field()
修改piepeline.py:用了內建imagespipeline儲存圖片到本地:
import pymongo from scrapy import Request from scrapy.exceptions import DropItem from scrapy.pipelines.images import ImagesPipeline class ImagePipeline(ImagesPipeline): def file_path(self, request, response=None, info=None): url = request.url file_name = url.split('/')[-1] return file_name def item_completed(self, results, item, info): image_paths = [x['path'] for ok, x in results if ok] if not image_paths: raise DropItem('Image Downloaded Failed') return item def get_media_requests(self, item, info): yield Request(item['url'])
最後修改setting:
ROBOTSTXT_OBEY = False #修改
ITEM_PIPELINES = {
'images360.pipelines.ImagePipeline': 300,
#'images360.pipelines.MongoPipeline': 301,
}
MAX_PAGE = 50
#MONGO_URI = '192.168.6.23'
#MONGO_DB = 'images360'
最後執行:
scrapy crawl images