新建一個scrapy工程

(python35) ubuntu@ubuntu:~/scrapy_project$ scrapy startproject huaban

新增一個spider

(python35) ubuntu@ubuntu:~/scrapy_project/huaban/huaban/spiders$ scrapy genspider huaban_pets huaban.com

目錄結構如下：

(python35) [email protected]:~/scrapy_project/huaban$ tree -I *.pyc
.
├── huaban
│   ├── __init__.py
 

│   ├── items.py
│   ├── middlewares.py
│   ├── pipelines.py
│   ├── __pycache__
│   ├── settings.py
│   └── spiders
│       ├── huaban_pets.py
│       ├── __init__.py
│       └── __pycache__
└── scrapy.cfg

編輯items.py檔案

# -*- coding: utf-8 -*-

import scrapy

class HuabanItem(scrapy.Item):
    img_url = scrapy.Field
 
()

編輯huaban_pets.py

# -*- coding: utf-8 -*-
import scrapy

class HuabanPetsSpider(scrapy.Spider):
    name = 'huaban_pets'
    allowed_domains = ['huaban.com']
    start_urls = ['http://huaban.com/favorite/pets/']
    def parse(self, response):
        for img_src in response.xpath('//*[@id="waterfall"]/div/a/img/@src'
 
).extract():
            item = HuabanmeinvItem()
            # 例如img_src為//img.hb.aicdn.com/223816b7fee96e892d20932931b15f4c2f8d19b315735-wgi1w2_fw236
            # 去掉後面的_fw236就為原圖
            item['img_url'] = 'http:' + img_src[:-6]
            yield item

編寫一箇中間鍵使用phantomj獲取網頁原始碼

在middlewares.py新增如下內容：

# -*- coding: utf-8 -*-

from scrapy import signals
from selenium import webdriver
from scrapy.http import HtmlResponse
from selenium.webdriver.common.desired_capabilities import DesiredCapabilities


class JSPageMiddleware(object):
    def process_request(self, request, spider):
        if spider.name == 'hbmeinv':
            # cap[".page.setting.resourceTimeout"] = 180
            # cap["chrome.page.setting.loadImage"] = False
            dcap = dict(DesiredCapabilities.PHANTOMJS)
            # 不載入圖片，爬頁面速度會快很多
            dcap["phantomjs.page.settings.loadImages"] = False
            browser = webdriver.PhantomJS(executable_path=r'/home/ubuntu/scrapy_project/huabanphantomjs',desired_capabilities=dcap)
            try:
                browser.get(request.url)
                return HtmlResponse(url=browser.current_url, body=browser.page_source,encoding='utf-8',request=request)
            except:
                print("get page failed!")
            finally:
                browser.quit()
        el   return

在pipelines.py中新增如下內容下載網頁圖片:

# -*- coding: utf-8 -*-

import urllib

class HuabanmeinvPipeline(object):
    def process_item(self, item, spider):
        url = item['img_url']
        urllib.request.urlretrieve(url, filename=r'/home/ubuntu/scrapy_project/huaban/image/%s.jpg' % url[url.rfind('/')+1:])
        return item

在setting.py中使用新增的中間鍵和設定訊息頭

DEFAULT_REQUEST_HEADERS = {
    'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
    'Accept-Language': 'zh-CN',
    'User-Agent' : 'Mozilla/4.0 (compatible; MSIE 5.5; Windows NT)'
}

# Enable or disable downloader middlewares
# See http://scrapy.readthedocs.org/en/latest/topics/downloader-middleware.html
DOWNLOADER_MIDDLEWARES = {
    #'huabanmeinv.middlewares.MyCustomDownloaderMiddleware': 543,
    'huabanmeinv.middlewares.JSPageMiddleware': 543,
}

# Configure item pipelines
# See http://scrapy.readthedocs.org/en/latest/topics/item-pipeline.html
ITEM_PIPELINES = {
   'huabanmeinv.pipelines.HuabanmeinvPipeline': 300,
}

開始爬取

ubuntu@ubuntu:~/scrapy_project/huaban/huaban/spiders$ scrapy runspider huaban_pets.py

爬取結束後，就可以在/home/ubuntu/scrapy_project/huaban/image目錄下看到爬取的圖片了，例如：