这可以通过Ray来实现,该系统允许您轻松地并行化和分发Python代码。
要并行化示例,需要使用@ray.remote装饰器定义函数,然后使用.remote调用它们。import ray
ray.init()
dir1 = 'C:\\folder1'
dir2 = 'C:\\folder2'
filename = 'test.txt'
addFiles = [25, 5, 15, 35, 45, 25, 5, 15, 35, 45]
# Define the functions.
# You need to pass every global variable used by the function as an argument.
# This is needed because each remote function runs in a different process,
# and thus it does not have access to the global variables defined in
# the current process.
@ray.remote
def func1(filename, addFiles, dir):
# func1() code here...
@ray.remote
def func2(filename, addFiles, dir):
# func2() code here...
# Start two tasks in the background and wait for them to finish.
ray.get([func1.remote(filename, addFiles, dir1), func2.remote(filename, addFiles, dir2)])
如果将同一个参数传递给两个函数,并且参数很大,则使用ray.put()是一种更有效的方法。这样可以避免将大参数序列化两次并创建其两个内存副本:largeData_id = ray.put(largeData)
ray.get([func1(largeData_id), func2(largeData_id)])
如果func1()和func2()返回结果,则需要按如下方式重写代码:ret_id1 = func1.remote(filename, addFiles, dir1)
ret_id2 = func1.remote(filename, addFiles, dir2)
ret1, ret2 = ray.get([ret_id1, ret_id2])
与multiprocessing模块相比,使用Ray有许多优点。尤其是,同一代码将在一台机器上以及在一组机器上运行。有关Ray的更多优点,请参见this related post。