Python Jupyter notebook 运行 multiprocessing 跑不了的问题

最近工作中为了解决python支持多核cpu，遇到一个Jupyter notebook跑不了multiprocessing的问题。

网上找了些multiprocessing的例子，Pycharm可以跑，但是在Jupyter notebook上跑了就只有In[*]，error log：

AttributeError: Can't get attribute 'task' on <module '__main__' <built-in>>

最后找到一个解决方案：把方法写到临时文件里，再读出来。

from multiprocessing import Pool
from functools import partial
import inspect

def parallal_task(func, iterable, *params):

    with open(f'./tmp_func.py', 'w') as file:
        file.write(inspect.getsource(func).replace(func.__name__, "task"))

    from tmp_func import task

    if __name__ == '__main__':
        func = partial(task, params)
        pool = Pool(processes=8)
        res = pool.map(func, iterable)
        pool.close()
        return res
    else:
        raise "Not in Jupyter Notebook"

def long_running_task(params, id):
    # Heavy job here
    return params, id

data_list = range(8)

for res in parallal_task(long_running_task, data_list, "a", 1, "b"):
    print(res) 

传送门：https://stackoverflow.com/questions/47313732/jupyter-notebook-never-finishes-processing-using-multiprocessing-python-3?r=SearchResults