Python中的多进程

multiprocessing模块

multiprocessing模块提供了一个 Process类来代表一个进程对象，下面的例子演示了启动一个子进程并等待其结束：

from multiprocessing import Process
import os

# 子进程要执行的代码
def run_proc(name):    
    print('Run child process %s (%s), Parent process is main (%s).' % (name, os.getpid(), os.getppid()))

if __name__=='__main__':
    print('Parent process %s.' % os.getpid())
    p = Process(target=run_proc, args=('test',))
    print('Child process will start.')
    p.start() # 启动进程
    p.join() # 等待子进程结束后再继续往下运行
    print('Child process end.') 

执行结果如下：

Parent process 928.
Process will start.
Run child process test (929), Parent process is main (928).
Process end.

928是父进程的pid，929是其子进程的pid。

Pool类

如果要启动大量的子进程，可以用进程池的方式批量创建子进程：

from multiprocessing import Pool
import os, time, random

def long_time_task(name):
    print('Run task %s (%s)...' % (name, os.getpid()))
    start = time.time()
    time.sleep(random.random() * 3)
    end = time.time()
    print('Task %s runs %0.2f seconds.' % (name, (end - start)))

if __name__=='__main__':
    print('Parent process %s.' % os.getpid())
    p = Pool()
    for i in range(5):
        p.apply_async(long_time_task, args=(i,))
    print('Waiting for all subprocesses done...')
    p.close()
    p.join()
    print('All subprocesses done.'

执行结果如下：

Parent process 5768.

Waiting for all subprocesses done...

Run task 0 (14964)...

Run task 1 (11252)...

Run task 2 (18272)...

Run task 3 (9340)...

Task 3 runs 0.11 seconds.

Run task 4 (9340)...

Task 0 runs 0.69 seconds.

Task 2 runs 2.19 seconds.

Task 4 runs 2.12 seconds.

Task 1 runs 2.47 seconds.

All subprocesses done.

第五个task 4要等待前面某个task完成后才执行，这是因为 Pool的默认大小在我的电脑内核数4，因此最多并行执行4个进程。如果 p = Pool(5)，依旧可以同时跑5个进程。

Parent process 12228.

Waiting for all subprocesses done...

Run task 0 (14928)...

Run task 1 (16848)...

Run task 2 (3272)...

Run task 3 (10792)...

Run task 4 (9300)...

Task 1 runs 0.11 seconds.

Task 2 runs 0.69 seconds.

Task 3 runs 1.13 seconds.

Task 0 runs 1.20 seconds.

Task 4 runs 1.60 seconds.

All subprocesses done.

附：Windows查看CPU：

CMD下：

wmic进入命令行系统管理执行脚本界面

通过cpu get *可以查看cpu的具体情况：

cpu核数 = NumberOfCores * NumberOfEnabledCore = NumberOfLogicalProcessors  

subprocess模块

subprocess模块可以让我们非常方便地启动一个子进程，然后控制其输入和输出。

通过调用subprocess.Popen 类可以创建并返回一个子进程，并在这个进程中执行指定的程序。

p = subprocess.Popen('ping www.python.org', shell=True)
p.wait()
等同于
r = subprocess.call('ping www.python.org', shell=True)

执行结果如下：

Pinging python.map.fastly.net [151.101.192.223] with 32 bytes of data:

Reply from 151.101.192.223: bytes=32 time=74ms TTL=52

Reply from 151.101.192.223: bytes=32 time=96ms TTL=52

Reply from 151.101.192.223: bytes=32 time=81ms TTL=52

Reply from 151.101.192.223: bytes=32 time=84ms TTL=52

Ping statistics for 151.101.192.223:

Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),

Approximate round trip times in milli-seconds:

Minimum = 74ms, Maximum = 96ms, Average = 83ms

进程间通信

Python的 multiprocessing模块包装了底层的机制，提供了 Queue、 Pipes等多种方式来交换数据。

from multiprocessing import Process, Queue, Pipe
import os, time, random

# 写数据进程执行的代码:
def write(q, p):
    print('Process to write: %s' % os.getpid())
    i = 1
    for value in ['A', 'B', 'C']:
        print('Put %s to queue...' % value)
        q.put(value)
        print('Put %s to pipe...' % i)
        p.send(i)
        time.sleep(random.random())
        i += 1
    p.close()

# 读数据进程执行的代码:
def read(q, p):
    print('Process to read: %s' % os.getpid())
    while True:
        value = q.get(True)
        print('Get %s from queue.' % value)
        print('Get %s from pipe.' % p.recv())

if __name__=='__main__':
    # 父进程创建Queue，并传给各个子进程：
    q = Queue()
    # 父进程创建Pipe：
    _out_pipe, _in_pipe = Pipe()
    pw = Process(target=write, args=(q, _in_pipe))
    pr = Process(target=read, args=(q, _out_pipe,))
    # 启动子进程pw，写入:
    pw.start()
    # 启动子进程pr，读取:
    pr.start()
    # 等待pw结束:
    pw.join()
    # pr进程里是死循环，无法等待其结束，只能强行终止:
    pr.terminate()

执行结果如下：

Process to write: 3960

Put A to queue...

Process to read: 10984

Put 1 to pipe...

Get A from queue.

Get 1 from pipe.

Put B to queue...

Put 2 to pipe...

Get B from queue.

Get 2 from pipe.

Put C to queue...

Put 3 to pipe...

Get C from queue.

Get 3 from pipe.

Ref：

https://blog.csdn.net/yeyingss/article/details/49385421

https://www.kancloud.cn/wizardforcel/liaoxuefeng/108671