Python multiprocessing、pool

multiprocessing

由于 Windows 没有 fork 调用，multiprocessing 模块提供了一个 Process 类来提供实现多进程的方式；

import os
from multiprocessing import Process

x1, x2 = 3, 7

def run_add(x1, x2):
    print("this is child process, the pid is %s" % os.getpid())
    print('x1 + x2 =', x1+x2)
    print("******the end*******")

if __name__ == '__main__':
    print("this is parent process, the pid is %s" % os.getpid())
    print('x1 * x2 =', x1*x2)
    p = Process(target=run_add, args=(x1, x2))
    p.start()
    print("******the end*******")
    p.join()
    # join()方法可以等待子进程结束后再继续往下运行，通常用于进程间的同步。

执行结果：

[root@syztoo ~]# python3 process.py
this is parent process, the pid is 7917
x1 * x2 = 21
******the end*******
this is child process, the pid is 7918
x1 + x2 = 10
******the end*******

Pool

如果要启动大量的子进程，可以用进程池的方式批量创建子进程：

from multiprocessing import Pool
import os, time, random

def long_time_task(name):
    print('Run task %s (%s)...' % (name, os.getpid()))
    start = time.time()
    time.sleep(random.random() * 3)
    end = time.time()
    print('Task %s runs %0.2f seconds.' % (name, (end - start)))

if __name__=='__main__':
    print('Parent process %s.' % os.getpid())
    p = Pool(4)
    for i in range(5):
        p.apply_async(long_time_task, args=(i,))
    print('Waiting for all subprocesses done...')
    p.close()
    p.join()
    print('All subprocesses done.')

执行结果如下：

Parent process 669.
Waiting for all subprocesses done...
Run task 0 (671)...
Run task 1 (672)...
Run task 2 (673)...
Run task 3 (674)...
Task 2 runs 0.14 seconds.
Run task 4 (673)...
Task 1 runs 0.27 seconds.
Task 3 runs 0.86 seconds.
Task 0 runs 1.41 seconds.
Task 4 runs 1.91 seconds.
All subprocesses done.

代码解读：

对 Pool 对象调用 join() 方法会等待所有子进程执行完毕，调用 join() 之前必须先调用 close()，调用 close() 之后就不能继续添加新的 Process 了。

请注意输出的结果，task 0，1，2，3是立刻执行的，而task 4要等待前面某个task完成后才执行，这是因为 Pool 的默认大小设置的是 4，因此，最多同时执行 4 个进程。这是 Pool有意设计的限制，并不是操作系统的限制。如果改成：

p = Pool(5)

就可以同时跑 5 个进程。

由于 Pool 的默认大小是 CPU 的核数，如果你拥有 8 核 CPU，你要提交至少 9 个子进程才能看到上面的等待效果。

---

from multiprocessing import Pool
import os,time,random

def run_add(x1,x2):
    time.sleep(random.random()*3)
    print('the pid is %s. x1 + x2 = %s' % (os.getpid(),x1+x2))

def run_sub(x1,x2):
    time.sleep(random.random()*3)
    print('the pid is %s. x1 - x2 = %s' % (os.getpid(),x1-x2))

def run_odd(x1,x2):
    time.sleep(random.random()*3)
    print('the pid is %s. x1 * x2 = %s' % (os.getpid(),x1*x2))

def run_div(x1,x2):
    time.sleep(random.random()*3)
    print('the pid is %s. x1 / x2 = %s' % (os.getpid(),x1/x2))

def run_rem(x1,x2):
    time.sleep(random.random()*3)
    print('the pid is %s. x1 %% x2 = %s' % (os.getpid(),x1%x2))


if __name__=='__main__':
    print('Parent process %s.' % os.getpid())
    x1,x2 = 7,3
    p = Pool(4)
    p.apply_async(run_add, args=(x1,x2))
    p.apply_async(run_sub, args=(x1,x2))
    p.apply_async(run_odd, args=(x1,x2))
    p.apply_async(run_div, args=(x1,x2))
    p.apply_async(run_rem, args=(x1,x2))
    p.close()
    p.join()
    print('All subprocesses done.')

执行结果：

[root@syztoo ~]# python3 pools.py 
Parent process 8124.
the pid is 8128. x1 / x2 = 2.3333333333333335
the pid is 8125. x1 + x2 = 10
the pid is 8128. x1 % x2 = 1
the pid is 8126. x1 - x2 = 4
the pid is 8127. x1 * x2 = 21
All subprocesses done.