Python非同步回撥(epoll)--典型的Event Loop例項
阿新 • • 發佈:2019-02-01
import socket from selectors import DefaultSelector, EVENT_WRITE, EVENT_READ selector = DefaultSelector() stopped = False urls_todo = {'/', '/1', '/2','/3', '/4', '/5', '/6', '/7', '/8'} class Crawler: def __init__(self, url): self.url = url self.sock = None self.response = b'' def fetch(self): self.sock = socket.socket() self.sock.setblocking(False) try: self.sock.connect(('example.com', 80)) except BlockingIOError: pass selector.register( self.sock.fileno(), EVENT_WRITE, self.connected ) def connected(self, key, mask): selector.unregister(key.fd) get = 'GET {0} HTTP/1.0\r\nHost:example.com \ \r\n\r\n'.format(self.url) self.sock.send(get.encode('ascii')) selector.register( key.fd, EVENT_READ, self.read_response ) def read_response(self, key, mask): globals stopped chunk = self.sock.recv(1024) if chunk: self.response += chunk else: selector.unregister(key.fd) urls_todo.remove(self.url) if not urls_todo: stopped = True def loop(): while not stopped: # 阻塞, 直到一個事件發生 events = selector.select() for event_key. event_mask in events: callback = event_key.data callback(event_key, event_mask) if __name__ == '__main__': import time start = time.time() for url in urls_todo: cralwer = Crawler(url) crawler.fetch() loop() print(time.time() - start)
- 建立Crawler 例項;
- 呼叫fetch方法,會建立socket連線和在selector上註冊可寫事件;
- fetch內並無阻塞操作,該方法立即返回;
- 重複上述3個步驟,將10個不同的下載任務都加入事件迴圈;
- 啟動事件迴圈,進入第1輪迴圈,阻塞在事件監聽上;
- 當某個下載任務EVENT_WRITE被觸發,回撥其connected方法,第一輪事件迴圈結束;
- 進入第2輪事件迴圈,當某個下載任務有事件觸發,執行其回撥函式;此時已經不能推測是哪個事件發生,因為有可能是上次connected裡的EVENT_READ先被觸發,也可能是其他某個任務的EVENT_WRITE被觸發;(此時,原來在一個下載任務上會阻塞的那段時間被利用起來執行另一個下載任務了
- 迴圈往復,直至所有下載任務被處理完成
- 退出事件迴圈,結束整個下載程式