多进程multiprocessing
multiprocessing is a package that supports spawning processes using an API similar to the threading module. The multiprocessing package offers both local and remote concurrency, effectively side-stepping the Global Interpreter Lock by using subprocesses instead of threads. Due to this, the multiprocessing module allows the programmer to fully leverage multiple processors on a given machine. It runs on both Unix and Windows.
To show the individual process IDs involved, here is an expanded example:
例子:
进程间通讯
不同进程间内存是不共享的,要想实现两个进程间的数据交换,可以用以下方法:
Queues
使用方法跟threading里的queue差不多,但是必须from multiprocessing import Queue 。其中Q是大写的,注意,要把生成的队列q作为参数传递给子进程(相当于克隆了一个q,当时其实底层就是两个q之间是再通过pickle序列化和反序列化再通信的,我们编程可以不关注底部通信),不然会报错,无法共享队列数据,另外,不能把一个线程的队列传给另外一个进程,只能在进程和进程间进行队列操作,或者线程和线程之间进行,再就是一个进程和里面的线程之间进行操作
注意:必须要把q作为参数传给另一个进程
Pipes(管道)
The Pipe() function returns a pair of connection objects connected by a pipe which by default is duplex (two-way). For example:
The two connection objects returned by Pipe() represent the two ends of the pipe. Each connection object has send() and recv() methods (among others). Note that data in a pipe may become corrupted if two processes (or threads) try to read from or write to the same end of the pipe at the same time. Of course there is no risk of corruption from processes using different ends of the pipe at the same time.
管道生成了之后会生成两个数据,例如:parent-conn,child-conn=Pipe(),管道两头一个放父进程,一个放子进程,放随无所谓,没有顺序,都可以,通过send和recv发和收消息
Managers
A manager object returned by Manager() controls a server process which holds Python objects and allows other processes to manipulate them using proxies.
A manager returned by Manager() will support (可以实现好多方法共享,默认给进程加锁了,实现多进程间共享数据)types list, dict, Namespace, Lock, RLock, Semaphore, BoundedSemaphore, Condition, Event, Barrier, Queue, Value and Array. For example,
进程同步
Without using the lock output from the different processes is liable to get all mixed up.这个锁存在的意义是屏幕共享,导致屏幕打印时不会乱,
进程池
进程池内部维护一个进程序列,当使用时,则去进程池中获取一个进程,如果进程池序列中没有可供使用的进进程,那么程序就会等待,直到进程池中有可用进程为止。
在windows系统上启动多进程,要if __name__=='__main__':(主要作用是将其下面的代码只在本模块执行,不能在其他模块调用运行下面的代码,可以不用from multiprocessing import freeze_support)
进程池中有两个方法:
apply 串行
apply_async 并行
例子中的callback好处:该函数直接在主进程执行,比如写日志的时候,直接是主进程和数据库连接,不需要每个子进程单独再去连接数据库,
注意:进程池只能先close,再join
selery 分布式任务队列
https://www.cnblogs.com/alex3714/p/6351797.html