python 多进程加速for循环示例,包括如何传参。

# SuperFastPython.com
# execute tasks in parallel in a for loop
from time import sleep
from random import random
from multiprocessing import Process


# execute a task
def task(arg, i, string_):
    print("arg=" + str(arg) + " i=" + str(i) + "string_" + string_)
    # generate a random value between 0 and 1
    value = random()
    # block for a fraction of a second
    sleep(value + arg[0])
    print(sum(arg))

    # report a message
    print(f'.done {arg[0]}, generated {value}', flush=True)


# protect the entry point
if __name__ == '__main__':
    pool_num = 3
    args_list = [[1,2,3,4], [5,6,7,8], [9,10,11,12]]
    string_ = "res"
    processes = [Process(target=task, args=(args_list[i], i, string_)) for i in range(pool_num)]
    # start all processes
    for process in processes:
        process.start()
    # wait for all processes to complete
    for process in processes:
        process.join()
    # report that all tasks are completed
    print('Done', flush=True)

过程就是定义几个Process,每个Process执行不同的task。task的区别在于for循环中不同的索引。另外各个Process如果输出文件,那么应写到不同的文件里避免不必要的麻烦。

args_list的由来:对于for循环而言,可以将整个长度进行切分成如上的args_list,再对各个Process传递不同的index值即可。如下所示:

split_size = int(len(accx_cuts) / pool_num)

print("allocate indexes")
args_list = []
for i in range(pool_num):
    data_list = [i * split_size + k for k in range(split_size)]
    print(i, data_list[0:5], data_list[-5:])
    args_list.append(data_list)

processes = [Process(target=task, args=(args_list[i], i, accx_cuts, accy_cuts, accz_cuts)) for i in range(pool_num)]

  • 0
    点赞
  • 9
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值