python並行化實現:利用multiprocessing
阿新 • • 發佈:2019-02-12
最近有個很繁瑣的需要提前計算資料指標的問題,很慢,於是就想到了並行化,之前沒有用過python進行並行化,搜了一下,受Caspar的譯文譯文啟發,原文在這裡,實現如下:
from multiprocessing import Pool
def compute(params):
'''
params:[param1,param2,param3...]
'''
# ...
pass
def get_params(dataset):
'''
return the list of params, each param is also a list
'''
# ...
params = []
for batch_num in xrange(num_batch):
param = []
param.append(param1)
param.append(param2)
param.append(param3)
params.append(param)
return params
if __name__ == '__main__':
dataset = 'xxxxxx'
params = get_params(dataset)
pool = Pool()
pool.map(compute, params)
pool.close()
pool.join()
這裡用列表來儲存多個引數。