python基础入门——并发编程

该部分在python的IO操作里有详细的介绍，这里主要讲并发编程的模块：threading模块、multiprocessing模块等内容

1、线程和进程

线程也叫轻量量级进程，它是一个基本的CPU执行单元，也是程序执行过程中的小单元。

进程就是一个程序在一个数据集上的一次动态执行过程。 进程一般由程序、数据集、进程控制块三部分组成。

线程是属于进程的，线程运行在进程空间内。

2、threading模块

参考：https://blog.csdn.net/zhangzheng0413/article/details/41728869/

join ()方法：主线程A中，创建了子线程B，并且在主线程A中调用了B.join()，那么，主线程A会在调用的地方等待，直到子线程B完成操作后，才可以接着往下执行，那么在调用这个线程时可以使用被调用线程的join方法

setDaemon()方法。主线程A中，创建了子线程B，并且在主线程A中调用了B.setDaemon(),这个的意思是，把主线程A设置为守护线程，这时候，要是主线程A执行结束了，就不管子线程B是否完成,一并和主线程A退出

方式一：传入函数

import threading
import time

def counter(n):
    cnt = 0
    for i in range(n):
        cnt += 1
        time.sleep(0.1)
        print(cnt)

# 传入一个函数
th = threading.Thread(target=counter,args=(10,)) # 注意这里的逗号不能少
th.start()
th.join() #t.join() ：逐个执行每个线程，执行完毕后继续往下执行
print('main thread task done')

方式二：继承类

import threading
import time
import random

def counter():
    cnt = 0
    for i in range(10000):
        for j in range(i):
            cnt +=j
    print('cnt = ',cnt)


class SubThread(threading.Thread):
    def __init__(self,name):
        threading.Thread.__init__(self,name=name)
        pass

    def run(self):
        i = 0
        while i <3:
            print(self.name,'counting ... \n')
            counter()
            print(self.name,'finish\n')
            i += 1

# 传入一个函数
th = SubThread('Thread-1')
th.setDaemon(True)  #setDaemon放在start方法之前,默认False，设置为True后，则主线程结束，子线程结束
th.start()
#th.join() #t.join() ：逐个执行每个线程，执行完毕后继续往下执行

print('main thread task done')

2.1、同步机制

参考：https://www.cnblogs.com/tkqasn/p/5700281.html

典型例子就是打印机

2.1.1 互斥锁Lock

from threading import Lock,Thread

lock = Lock()
some_var = 0

class IncrementThread(Thread):
    def run(self):
        global some_var
        #lock.acquire(True) #提供了两种应用方式 acquire release
        with lock: # with lock 方式
            read_value = some_var
            print('some_var in %s is %d'%(self.name,read_value))
            some_var = read_value + 1
            print('some_var in %s after increment is %d'%(self.name,some_var))
        #lock.release()

def use_increment_thread():
    threads =[]
    for i in range(50):
        t = IncrementThread()
        threads.append(t)
        t.start()
    for t in threads:
        t.join()
    print('after 50 modification ,some_var shoule have become 50')
    print('after 50 modifications,some_var is %d'%(some_var,))

use_increment_thread()

2.1.2 可重入锁RLock

参考：https://blog.csdn.net/davidsu33/article/details/51385965

           https://blog.csdn.net/ybdesire/article/details/80294638

RLock允许在同一线程中被多次acquire。而Lock却不允许这种情况。注意：如果使用RLock，那么acquire和release必须成对出现，即调用了n次acquire，必须调用n次的release才能真正释放所占用的琐。

import threading


print('lock acquire')
lock = threading.Lock() #线程锁死
lock.acquire()
lock.acquire()
lock.release()
lock.release()
print('done')

print('lock acquire')
lock = threading.RLock() #线程不锁死
lock.acquire()
lock.acquire()
lock.release()
lock.release()
print('done')

2.1.3 Condition条件同步机制

一个线程等待特定条件，而另一个线程发出特定条件满足的信号。

import multiprocessing
import random
import time

item = 0


class producer(multiprocessing.Process):
    def __init__(self, n):
        multiprocessing.Process.__init__(self)
        self.n = n

    def run(self):
        for i in range(5):
            # item = random.randint(0,256)
            global item
            item = item + 1
            self.n.value = item
            print('<--process producer : item %d appended to queue %s>' % (self.n.value, self.name))
            time.sleep(1)


class consumer(multiprocessing.Process):
    def __init__(self, n):
        multiprocessing.Process.__init__(self)
        self.n = n

    def run(self):
        time.sleep(1)
        while True:
            time.sleep(1)
            try:
                item = self.n.value
                print('<--process consumer : item %d popped from queue %s>' % (item, self.name))
            except Exception as e:
                print('the queue is empty,process consumer exit')
                break
        time.sleep(1)


if __name__ == '__main__':
    num = multiprocessing.Value('d', 0.0)
    process_producer = producer(num)
    process_consumer = consumer(num)
    process_producer.start()
    process_consumer.start()
    process_producer.join()
    process_consumer.join()

import threading,time

class Seeker(threading.Thread):
    def __init__(self,cond,name):
        super(Seeker, self).__init__()
        self.cond = cond
        self.name = name

    def run(self):
        print('i am here 1')
        time.sleep(1) # 这里的sleep()是为了保证hinder首先走到wait()那里阻塞住，然后等seeker的notify()
        print('i am here 3') #在没有wait的过程中，可以认为两个线程是同时进行的
        self.cond.acquire()   # with self.cond:
        print(self.name + ': 我已经把眼睛蒙上了')
        self.cond.notify()
        self.cond.wait()

        print(self.name + ': 我找到你了 ！！')
        self.cond.notify()
        print('i an here 6')
        self.cond.release()
        print('i an here 7')

        time.sleep(1)
        print('i an here 10')

        print(self.name + ': 我赢了')

class Hinder(threading.Thread):
    def __init__(self,cond,name):
        super(Hinder,self).__init__()
        self.cond = cond
        self.name = name
    def run(self):
        print('i am here 2')
        self.cond.acquire() # with self.cond:
        print('i am here 4')
        self.cond.wait()
        print('i am here 5')
        print(self.name + ': 我已经藏好了，来找我吧')
        self.cond.notify()
        self.cond.wait()

        print('i am here 8')

        self.cond.release()
        print('i am here 9')
        print(self.name + ': 被你找到了')

cond = threading.Condition()
seeker = Seeker(cond,'seeker')
hider = Hinder(cond,'hider')

hider.start()  # 谁先start(),谁就先第一个输出，并同时运行
seeker.start()

输出：
i am here 2
i am here 4
i am here 1
i am here 3
seeker: 我已经把眼睛蒙上了
i am here 5
hider: 我已经藏好了，来找我吧
seeker: 我找到你了 ！！
i an here 6
i an here 7
i am here 8
i am here 9
hider: 被你找到了
i an here 10
seeker: 我赢了

2.1.4 Event 事件同步机制

     Event（事件）是最简单的线程通信机制之一：一个线程通知事件，其他线程等待事件。Event内置了一个初始为False的标志，当调用set()时设为True，调用clear()时重置为 False。wait()将阻塞线程至等待阻塞状态。Event其实就是一个简化版的 Condition。Event没有锁，无法使线程进入同步阻塞状态。

import threading
import random
import time

class VehicleThread(threading.Thread):

    def __init__(self,threadName,event):

        threading.Thread.__init__(self,name=threadName)
        self.threadEvent = event

    def run(self):
        time.sleep(random.randrange(1,10))
        print('%s arrived at %s\n' %(self.getName(),time.ctime(time.time())))
        self.threadEvent.wait() # 如果标志为True将立即返回，否则阻塞线程至等待阻塞状态，等待其他线程调用set()
        print('%s passed through intersection at %s\n'%(self.getName(),time.ctime(time.time())))

greenLight = threading.Event()
vehicleThreads = []

for i in range(1,5):
    vehicleThreads.append(VehicleThread('Vehicle' + str(i),greenLight))

for vehicle in vehicleThreads:
    vehicle.start()

while threading.activeCount() > 1: #threading.activeCount(): 返回正在运行的线程数量
    greenLight.clear() #将标志设为False
    print('Red light! at',time.ctime(time.time()))
    time.sleep(3)

    print('green light at ',time.ctime(time.time()))
    greenLight.set() #将标志设为True 
    time.sleep(1)

2.1.5 信号量

信号量通过一个计数器控制对共享资源的访问（例如咖啡机），当此线程不再需要访问共享资源时，它释放该通行证，这导致信号量的计数递增，如果另一个线程等待通行证，则那个线程将在那时获取通行证。

import time
from random import random
from threading import Thread,Semaphore
sema = Semaphore(5)
def foo(tid):
    with sema:
        print('{} acquire sema'.format(tid))
        wt = random() * 2
        time.sleep(wt)
    print('{} release sema'.format(tid))

threads = []
for i in range(5):
    t = Thread(target=foo,args=(i,))
    threads.append(t)
    t.start()
for t in threads:
    t.join()

2.1.6 GIL 全局解释器锁

参考：https://www.cnblogs.com/cjaaron/p/9166538.html （写的非常详细）

任意时刻，只有一个线程在解释器中运行，多线程在Python中只能交替执行，即使100个线程跑在100核CPU上，也只能用到1个核。Python的多线程在多核CPU上，只对于IO密集型计算产生正面效果；而当有至少有一个CPU密集型线程存在，那么多线程效率会由于GIL而大幅下降。

3、multiprocessing模块

multiprocess的引入会增加程序实现时线程间数据通讯和同步的困难

与threading模块一样， 创建类有两种方式，方式一：传入函数

import multiprocessing
import time

def foo(i):
    print('called function in process: %s'% i)
    time.sleep(5)
    return


if __name__ == '__main__':  # 不能缺少，否则会报错
    Process_jobs = []
    for i in range(5):
        p = multiprocessing.Process(target=foo,args=(i,))
        Process_jobs.append(p)
        p.start()

    for p in Process_jobs:
        p.join()

import multiprocessing
import time

def foo():
    name = multiprocessing.current_process().name
    print('Starting %s \n' %name)
    time.sleep(3)
    print('Exiting %s \n' %name)



if __name__ == '__main__':  # 不能缺少，否则会报错

    background_process = multiprocessing.Process(name='background_process',target=foo)
    background_process.daemon =True  #主线程结束，子线程结束
    No_background_process = multiprocessing.Process(name='No_background_process',target=foo)
    No_background_process.daemon = False


    No_background_process.start()
    background_process.start()

输出：
Starting No_background_process 

Exiting No_background_process 

杀死进程

import multiprocessing
import time

def foo():
    name = multiprocessing.current_process().name
    print('Starting %s \n' %name)
    time.sleep(0.1)
    print('Exiting %s \n' %name)



if __name__ == '__main__':  # 不能缺少，否则会报错

     p= multiprocessing.Process(name='process-#test#',target=foo)
     print('process before execution:',p,p.is_alive())
     p.start()
     print('process running:',p,p.is_alive())
     p.terminate()  #terminate 关闭进程,不会立即关闭,有个等着操作系统去关闭这个进程的时间,所以is_alive立刻查看的结果可能还是存活，但是稍微等一会，就被关掉了
     print('process terminated:',p,p.is_alive())
     p.join()
     print('process joined:',p,p.is_alive())
     print('process exit code:',p.exitcode)

输出：
process before execution: <Process(process-#test#, initial)> False
process running: <Process(process-#test#, started)> True
process terminated: <Process(process-#test#, started)> True
process joined: <Process(process-#test#, stopped[SIGTERM])> False
process exit code: -15

方式二：继承类

import multiprocessing

class MyProcess(multiprocessing.Process):
    def run(self):
        print(self.name)  #默认生成的name，在multiprocessing.Process初始化函数中
        print('called run method in process: %s' %self.name)
        return
if __name__ == '__main__':
    jobs = []
    for i in range(5):
        p = MyProcess()
        jobs.append(p)
        p.start()
        p.join()

输出：
MyProcess-1
called run method in process: MyProcess-1
MyProcess-2
called run method in process: MyProcess-2
MyProcess-3
called run method in process: MyProcess-3
MyProcess-4
called run method in process: MyProcess-4
MyProcess-5
called run method in process: MyProcess-5

3.1 进程间通信

参考：https://www.cnblogs.com/qing-chen/p/7688343.html

3.1.1 共享内存

参考：https://blog.csdn.net/sangky/article/details/82467337 （写的非常详细）

from multiprocessing import Process,Value,Array

def f(n,a):
    n.value = 3.1415927
    for i in range(len(a)):
        a[i] = -a[i]

if __name__ == '__main__':
    num = Value('d',0.0)
    arr = Array('i',range(10))

    p = Process(target=f,args=(num,arr))
    p.start()
    p.join()

    print(num.value)
    print(arr[:])

输出：
3.1415927
[0, -1, -2, -3, -4, -5, -6, -7, -8, -9]

3.1.2 队列

Queue()方法相当于第三方，把进程A的数据序列化后传给进程B 反序列化得到数据。并不是一个共享的变量。而是实现了数据的传递。

import multiprocessing
import random
import time

item = 0
class producer(multiprocessing.Process):
    def __init__(self,queue):
        multiprocessing.Process.__init__(self)
        self.queue = queue
    def run(self):
        for i in range(5):
            #item = random.randint(0,256)
            global item
            item = item + 1
            self.queue.put(item)
            print('<--process producer : item %d appended to queue %s>'%(item,self.name))
            time.sleep(1)


class consumer(multiprocessing.Process):
    def __init__(self,queue):
        multiprocessing.Process.__init__(self)
        self.queue = queue
    def run(self):
        time.sleep(3)
        while True:
            time.sleep(2)
            try:
                item = self.queue.get()
                print('<--process consumer : item %d popped from queue %s>'%(item,self.name))
            except Exception as e:
                print('the queue is empty,process consumer exit')
                break
        time.sleep(1)

if __name__ == '__main__':
    queue = multiprocessing.Queue()
    queue.put(-1)
    process_producer = producer(queue)
    process_consumer = consumer(queue)
    process_producer.start()
    process_consumer.start()
    process_producer.join()
    process_consumer.join()

3.1.3 管道

类似socket通信

from multiprocessing import Process,Pipe
class producer(Process):
    def __init__(self,pipe):
        Process.__init__(self)
        self.pipe = pipe
    def run(self):
        print('i am here 1')
        self.pipe.send('consumer words 1')
        print('i am here 2')
        print('consumer receied',self.pipe.recv())

class consumer(Process):
    def __init__(self,pipe):
        Process.__init__(self)
        self.pipe = pipe
    def run(self):
        print('i am here 3')
        print('producer receied',self.pipe.recv())
        print('i am here 4')
        self.pipe.send('producer words 2')

if __name__ == '__main__':
    pipe = Pipe()
    p = producer(pipe[0])
    c = consumer(pipe[1])
    p.daemon = c.daemon =True
    p.start()
    c.start()
    p.join()
    c.join()
    print('ended')

输出：
i am here 1
i am here 2
i am here 3
producer receied consumer words 1
i am here 4
consumer receied producer words 2
ended

3.2 进程池

Pool可以提供指定数量的进程供用户调用，当有新的请求提交到pool中时 ， 如果池还没有满，那么就会创建一个新的进程用来执行该请求； 但如果池中的进程数已经达到规定大值，那么该请求就会等待 ，直到池中有进程结束，才会创建新的进程来它。

from multiprocessing import Lock,Pool
import time

def function(index):
    print('start process:',index)
    time.sleep(3)
    print('end process',index)


if __name__ == '__main__':
    pool = Pool(processes=3)
    for i in range(4):
        pool.apply(function,(i,))

    print('started process')
    pool.close()
    pool.join()
    print('done')

输出：
start process: 0
end process 0
start process: 1
end process 1
start process: 2
end process 2
start process: 3
end process 3
started process
done