python3中線程池
阿新 • • 發佈:2017-06-06
pen stderr blog 成了 pool hcl 換上 err urn
1.在使用多線程處理任務時也不是線程越多越好,由於在切換線程的時候,需要切換上下文環境,依然會造成cpu的大量開銷。為解決這個問題,線程池的概念被提出來了。預先創建好一個較為優化的數量的線程,讓過來的任務立刻能夠使用,就形成了線程池。在python中,沒有內置的較好的線程池模塊,需要自己實現或使用第三方模塊。下面是一個簡單的線程池:
import threading,time,os,queue class ThreadPool(object): def __init__(self,maxsize): self.maxsize = maxsize self._q = queue.Queue(self.maxsize) for i in range(self.maxsize): self._q.put(threading.Thread) def getThread(self): return self._q.get() def addThread(self): self._q.put(threading.Thread) def fun(num,p): print(‘this is thread [%s]‘%num) time.sleep(1) p.addThread() if __name__ == ‘__main__‘: pool = ThreadPool(2) for i in range(103): t = pool.getThread() a = t(target = fun,args = (i,pool)) a.start()
2.利用線程池和paramiko實現對遠程服務器的訪問獲取到相關信息:(自己寫的例子,比較low)
import paramiko,threading import queue class ThreadPool(object): def __init__(self,maxsize): self.maxsize = maxsize self._q = queue.Queue(self.maxsize) for i in range(self.maxsize): self._q.put(threading.Thread) def getThread(self): return self._q.get() def addThread(self): self._q.put(threading.Thread) def ssh_fun(ip,user,password,pool): try: ssh = paramiko.SSHClient() ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy()) ssh.connect(ip, 22, user, password) stdin, stdout, stderr = ssh.exec_command(‘hostname‘) info = stdout.read().decode().strip() print(‘IP:%s hostname:%s‘%(ip,info)) ssh.close() except Exception: print(‘sorry I can`t connect this server [%s]‘%ip) pool.addThread() if __name__ == ‘__main__‘: t_list = [] pool = ThreadPool(2) with open(‘aaa‘,‘r+‘,encoding=‘utf-8‘) as f: for line in f: split = line.split() ip,user,password = split[0],split[1],split[2] th = pool.getThread() t = th(target=ssh_fun,args=(ip,user,password,pool)) t.start() t_list.append(t) for i in t_list: i.join()
在這裏我為了測試線程池中只有兩個線程,並且我這個是讀取aaa文件的,這個文件中包含用戶名和密碼等相關信息,樣子如下(其實可以把這些放進數據庫中,使用python從數據庫中進行讀取):
192.168.32.167 root 111111 192.168.32.110 root 111111 192.168.32.120 root 111111 192.168.32.150 root 111111
而最後執行的效果如下:
IP:192.168.32.167 hostname:ns.root
sorry I can`t connect this server [192.168.32.110]
IP:192.168.32.150 hostname:localhost.localdomain
sorry I can`t connect this server [192.168.32.120]
python3中線程池