安裝splash

1、安裝docker（參考：mac安裝docker）
2、安裝splash

docker pull scrapinghub/splash  # 安裝

docker run -p 8050:8050 scrapinghub/splash  # 執行

程式碼示例

import requests
import time
from scrapy import Selector


def timer(func):
    def inner(*args):
        start = time.time()
        response = func(*args)
        print("time: %s"
 
 % (time.time() - start))
        return response
    return inner


@timer
def use_request(url):
     return requests.get(url)


@timer
def use_splash(url):
    splash_url = "http://localhost:8050/render.html"

    args = {
        "url": url,
        "timeout": 5,
        "image": 0
    }

    return requests.get(splash_url, params=args)


if
 
 __name__ == '__main__':

    url = "http://quotes.toscrape.com/js/"

    r1 = use_request(url)
    sel1 = Selector(r1)
    text = sel1.css(".quote .text::text").extract_first()
    print(text)

    r2 = use_splash(url)
    sel2 = Selector(r2)
    text = sel2.css(".quote .text::text").extract_first()
    print(text)

"""
time: 0.632809877396
None

time: 0.685022830963
“The world as we have created it is a process of our thinking. 
    It cannot be changed without changing our thinking.”
"""
 

通過測試，發現需要splash對網頁進行了渲染，獲取到了資料，而且速度還很快

args引數說明：
url: 需要渲染的頁面地址
timeout: 超時時間
proxy：代理
wait：等待渲染時間
images: 是否下載，預設1（下載）
js_source: 渲染頁面前執行的js程式碼