（Python）在Spyder中使用multiprocessing多进程遇到的问题

by 今天不飞了

为了提高计算效率，开始学习multiprocessing。在学习练习阶段，各种示例搞得风生水起，一到实战场景复杂就出“BUG”了（进程一直无法结束）。查阅各种资料都没解决方案，最后发现压根不是代码的错，而是Spyder的锅。今天纪录在此，希望对在做同样事情的你有所帮助。

遇到的问题

写了一个简单的多进程，如下

import multiprocessing as mp
import time

def kernel1(name, data):
    print(name+str(data))
    result = data*2
    return {name: result}

if __name__ == '__main__':

    # data
    data = {'task1': 1,
            'task2': 2,
            'task3': 3,
            'task4': 4}

    # cpus
    coresNum = int(mp.cpu_count())
    print("CPUs :" + str(coresNum))
    
    # multiprocessing
    ta = time.time()
    pool = mp.Pool(coresNum)
    results = [pool.apply_async(kernel1, 
            args=(name, param)) for name, param in data.items()]
    results = [p.get() for p in results]
    tb = time.time()
    print("it took " + str(round(tb-ta,4)))
    
    # close
    pool.close()
    
    # print
    print(results)

运行没毛病，瞬间出结果

然后我试图加入自己的库函数，修改后如下

import multiprocessing as mp
import time
from myfunc import kernel2 # 这里这里，多加了这一句而已

def kernel1(name, data):
    print(name+str(data))
    result = data*2
    return {name: result}

if __name__ == '__main__':

    # data
    data = {'task1': 1,
            'task2': 2,
            'task3': 3,
            'task4': 4}

    # cpus
    coresNum = int(mp.cpu_count())
    print("CPUs :" + str(coresNum))
    
    # multiprocessing
    ta = time.time()
    pool = mp.Pool(coresNum)
    results = [pool.apply_async(kernel1, 
            args=(name, param)) for name, param in data.items()]
    results = [p.get() for p in results]
    tb = time.time()
    print("it took " + str(round(tb-ta,4)))
    
    # close
    pool.close()
    
    # print
    print(results)

仅仅是import了自己写的函数，甚至都没调用，再次运行……
结果四个进程仿佛被“砸瓦鲁多”，一点反应都没有了（在如下画面停留2分钟，不得不结束进程）……

---

解决方案

做了好多尝试之后发现是Spyder的锅……
改用命令行直接运行，完美解决……

然后把代码改成了如下，

import multiprocessing as mp
import time
from myfunc import kernel2 # 这里这里，多加了这一句而已

if __name__ == '__main__':

    # data
    data = {'task1': 1,
            'task2': 2,
            'task3': 3,
            'task4': 4}

    # cpus
    coresNum = int(mp.cpu_count())
    print("CPUs :" + str(coresNum))
    
    # multiprocessing
    ta = time.time()
    pool = mp.Pool(coresNum)
    results = [pool.apply_async(kernel2, 
            args=(name, param)) for name, param in data.items()]
    results = [p.get() for p in results]
    tb = time.time()
    print("it took " + str(round(tb-ta,4)))
    
    # close
    pool.close()
    
    # print
    print(results)

其中kernel2就是kernel1，只不过单独放在myfunc.py中

其他

更多问题，欢迎留言