Selenium基於webkit實現爬蟲功能

http://www.cnblogs.com/luxiaojun/p/6144748.html

https://www.cnblogs.com/chenqingyang/p/3772673.html

現在headless chrome替代 PhantomJS

https://zhuanlan.zhihu.com/p/27100187

from selenium import webdriver
from selenium.webdriver.common.desired_capabilities import DesiredCapabilities
import time
import io

dcap = dict(DesiredCapabilities.PHANTOMJS)  #設置userAgent
dcap["phantomjs.page.settings.userAgent"] = ("Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:25.0) Gecko/20100101 Firefox/25.0 ")
 
obj = webdriver.PhantomJS(executable_path=‘C:\Program Files (x86)\Microsoft Visual Studio\Shared\Python36_64\Scripts\phantomjs.exe‘,desired_capabilities=dcap) #加載網址
obj.get(‘http://chart.icaile.com/sd11x5.php‘)#打開網址


#time.sleep(10)
pageSource = obj.page_source
print(pageSource)

obj.quit() 

獲取的網頁內容後，可以使用beautifulsoup來分析

https://cuiqingcai.com/1319.html

Selenium 獲取動態js的網頁