by 今天不飞了
为了提高计算效率,开始学习multiprocessing。在学习练习阶段,各种示例搞得风生水起,一到实战场景复杂就出“BUG”了(进程一直无法结束)。查阅各种资料都没解决方案,最后发现压根不是代码的错,而是Spyder的锅。今天纪录在此,希望对在做同样事情的你有所帮助。
遇到的问题
写了一个简单的多进程,如下
import multiprocessing as mp
import time
def kernel1(name, data):
print(name+str(data))
result = data*2
return {name: result}
if __name__ == '__main__':
# data
data = {'task1': 1,
'task2': 2,
'task3': 3,
'task4': 4}
# cpus
coresNum = int(mp.cpu_count())
print("CPUs :" + str(coresNum))
# multiprocessing
ta = time.time()
pool = mp.Pool(coresNum)
results = [pool.apply_async(kernel1,
args=(name, param)) for name, param in data.items()]
results = [p.get() for p in results]
tb = time.time()
print("it took " + str(round(tb-ta,4)))
# close
pool.close()
# print
print(results)
运行没毛病,瞬间出结果
然后我试图加入自己的库函数,修改后如下
import multiprocessing as mp
import time
from myfunc import kernel2 # 这里这里,多加了这一句而已
def kernel1(name, data):
print(name+str(data))
result = data*2
return {name: result}
if __name__ == '__main__':
# data
data = {'task1': 1,
'task2': 2,
'task3': 3,
'task4': 4}
# cpus
coresNum = int(mp.cpu_count())
print("CPUs :" + str(coresNum))
# multiprocessing
ta = time.time()
pool = mp.Pool(coresNum)
results = [pool.apply_async(kernel1,
args=(name, param)) for name, param in data.items()]
results = [p.get() for p in results]
tb = time.time()
print("it took " + str(round(tb-ta,4)))
# close
pool.close()
# print
print(results)
仅仅是import了自己写的函数,甚至都没调用,再次运行……
结果四个进程仿佛被“砸瓦鲁多”,一点反应都没有了(在如下画面停留2分钟,不得不结束进程)……
解决方案
做了好多尝试之后发现是Spyder的锅……
改用命令行直接运行,完美解决……
然后把代码改成了如下,
import multiprocessing as mp
import time
from myfunc import kernel2 # 这里这里,多加了这一句而已
if __name__ == '__main__':
# data
data = {'task1': 1,
'task2': 2,
'task3': 3,
'task4': 4}
# cpus
coresNum = int(mp.cpu_count())
print("CPUs :" + str(coresNum))
# multiprocessing
ta = time.time()
pool = mp.Pool(coresNum)
results = [pool.apply_async(kernel2,
args=(name, param)) for name, param in data.items()]
results = [p.get() for p in results]
tb = time.time()
print("it took " + str(round(tb-ta,4)))
# close
pool.close()
# print
print(results)
其中kernel2就是kernel1,只不过单独放在myfunc.py中
其他
更多问题,欢迎留言