1. 程式人生 > >Python3之協程爬蟲應用

Python3之協程爬蟲應用

from gevent import monkey;

monkey.patch_all()
import gevent, requests


def get_page(url):
    ret = requests.get(url).text
    with open('b.txt', 'a', encoding='utf-8') as f:
        f.write('%s-%s\n' % (url, len(ret)))


# g1 = gevent.spawn(get_page, 'http://www.openstack.org')
g2 = gevent.spawn(get_page, 'https://www.python.org'
) g3 = gevent.spawn(get_page, 'http://www.sina.com.cn/') gevent.joinall([g2, g3])

所用時間是網路延遲最長還有寫檔案最耗時的那個任務的時間!!!