python3.6 多程序+協程的配合 提升爬蟲效率?
阿新 • • 發佈:2019-01-10
上篇部落格協程asyncio+aiohttp模組非同步採集資料,效率比多執行緒或多程序高很多。是否還能再優化,利用多程序+協程將效率翻倍呢?
程式碼:
(參照 L瑜的文章:<python中多程序+協程的使用以及為什麼要用它>寫了如下測試,非同步部分沒有用gevent而是asyncio)
import asyncio import threading import multiprocessing from multiprocessing import Queue ,Pool,Process #import aiohttp
import os
async def hello(name): print('hello {}{}**********{}'.format(name,os.getpid(),threading.current_thread())) #await asyncio.sleep(int(name)) await asyncio.sleep(1) print('end:{}{}'.format(name,os.getpid())) def process_start(*namelist): tasks=[] loop=asyncio.get_event_loop() for name in namelist: tasks.append(asyncio.ensure_future(hello(name))) loop.run_until_complete(asyncio.wait(tasks)) deftask_start(namelist): i=0 lst=[] flag=10 while namelist: i+=1 l=namelist.pop() lst.append(l) if i==flag: p=Process(target=process_start,args=lst) p.start() #p.join() lst=[] i=0 if namelist!=[]: p=Process(target=process_start,args=lst) p.start() #p.join() if __name__=='__main__': namelist=list('0123456789'*10) task_start(namelist) ''' loop=asyncio.get_event_loop() tasks=[] namelist=list('0123456789'*10) for i in namelist: tasks.append(asyncio.ensure_future(hello(i))) loop.run_until_complete(asyncio.wait(tasks))'''
誰有多核的電腦執行下哈 我這單核的跑不出來效率