最近工作中为了解决python支持多核cpu,遇到一个Jupyter notebook跑不了multiprocessing的问题。
网上找了些multiprocessing的例子,Pycharm可以跑,但是在Jupyter notebook上跑了就只有In[*],error log:
AttributeError: Can't get attribute 'task' on <module '__main__' <built-in>>
最后找到一个解决方案:把方法写到临时文件里,再读出来。
from multiprocessing import Pool
from functools import partial
import inspect
def parallal_task(func, iterable, *params):
with open(f'./tmp_func.py', 'w') as file:
file.write(inspect.getsource(func).replace(func.__name__, "task"))
from tmp_func import task
if __name__ == '__main__':
func = partial(task, params)
pool = Pool(processes=8)
res = pool.map(func, iterable)
pool.close()
return res
else:
raise "Not in Jupyter Notebook"