python并行处理任务,在python中并行执行任务

I am using python 2.7, I have some code that looks like this:

task1()

task2()

task3()

dependent1()

task4()

task5()

task6()

dependent2()

dependent3()

The only dependencies here are as follows: dependent1 needs to wait for tasks1-3, dependent2 needs to wait for tasks 4-6 and dependent3 needs to wait for dependents1-2... The following would be okay: running the whole 6 tasks first in parallel, then the first two dependents in parallel.. then the final dependent

I prefer to have as much tasks as possible running in parallel, I've googled for some modules but I was hoping to avoid external libraries, and not sure how the Queue-Thread technique can solve my problem (maybe someone can recommend a good resource?)

解决方案

The builtin threading.Thread class offers all you need: start to start a new thread and join to wait for the end of a thread.

import threading

def task1():

pass

def task2():

pass

def task3():

pass

def task4():

pass

def task5():

pass

def task6():

pass

def dep1():

t1 = threading.Thread(target=task1)

t2 = threading.Thread(target=task2)

t3 = threading.Thread(target=task3)

t1.start()

t2.start()

t3.start()

t1.join()

t2.join()

t3.join()

def dep2():

t4 = threading.Thread(target=task4)

t5 = threading.Thread(target=task5)

t4.start()

t5.start()

t4.join()

t5.join()

def dep3():

d1 = threading.Thread(target=dep1)

d2 = threading.Thread(target=dep2)

d1.start()

d2.start()

d1.join()

d2.join()

d3 = threading.Thread(target=dep3)

d3.start()

d3.join()

Alternatively to join you can use Queue.join to wait for the threads end.