引入进程池
在学习线程池之前,我们先看一个例子
from multiprocessing import Process
import time
def task(name):
print("name",name)
time.sleep(1)
if __name__ == "__main__":
start = time.time()
p1 = Process(target=task,args=("safly1",))
p2 = Process(target=task, args=("safly2",))
p3 = Process(target=task, args=("safly3",))
p1.start()
p2.start()
p3.start()
p1.join()
p2.join()
p3.join()
print("main")
end = time.time()
print(end- start)
输出如下:
name safly1
name safly2
name safly3
main
1.2071197032928467
以上的方式是一个个创建进程,这样的耗费时间才1秒多,虽然高效,但是有什么弊端呢?
如果并发很大的话,会给服务器带来很大的压力,所以引入了进程池的概念
使用ProcessPoolExecutor进程池
Python3.2开始,标准库为我们提供了concurrent.futures模块,它提供了ThreadPoolExecutor和ProcessPoolExecutor两个类,实现了对threading和multiprocessing的进一步抽象,对编写线程池/进程池提供了直接的支持。
通过ProcessPoolExecutor 来做示例。
我们来看一个最简单的进程池
from concurrent.futures import ProcessPoolExecutor
import time
def task(name):
print("name",name)
time.sleep(1)
if __name__ == "__main__":
start = time.time()
ex = ProcessPoolExecutor(2)
for i in range(5):
ex.submit(task,"safly%d"%i)
ex.shutdown(wait=True)
print("main")
end = time.time()
print(end - start)
输出如下:
E:\python\python_sdk\python.exe "E:/python/py_pro/4 进程池.py"
name safly0
name safly1
name safly2
name safly3
name safly4
main
3.212218999862671
简单解释下:
ProcessPoolExecutor(2)创建一个进程池,容量为2,循环submit出5个进程,然后就在线程池队列里面,执行多个进程,ex.shutdown(wait=True)意思是进程都执行完毕,在执行主进程的内容
使用shutdown
ex.shutdown(wait=True)是进程池内部的进程都执行完毕,才会关闭,然后执行后续代码
如果改成false呢?看如下代码
from concurrent.futures import ProcessPoolExecutor
import time
def task(name):
print("name",name)
time.sleep(1)
if __name__ == "__main__":
start = time.time()
ex = ProcessPoolExecutor(2)
for i in range(5):
ex.submit(task,"safly%d"%i)
ex.shutdown(wait=False)
print("main")
end = time.time()
print(end - start)
输出如下:
main
0.01500844955444336
name safly0
name safly1
name safly2
name safly3
name safly4
使用submit同步调用
同步调用:提交/调用一个任务,然后就在原地等着,等到该任务执行完毕拿到结果,再执行下一行代码
from concurrent.futures import ProcessPoolExecutor
import time, random, os
def piao(name, n):
print('%s is piaoing %s' % (name, os.getpid()))
time.sleep(1)
return n ** 2
if __name__ == '__main__':
p = ProcessPoolExecutor(2)
start = time.time()
for i in range(5):
res=p.submit(piao,'safly %s' %i,i).result() #同步调用
print(res)
p.shutdown(wait=True)
print('主', os.getpid())
stop = time.time()
print(stop - start)
输出如下:
E:\python\python_sdk\python.exe "E:/python/py_pro/4 进程池.py"
safly 0 is piaoing 12996
0
safly 1 is piaoing 14044
1
safly 2 is piaoing 12996
4
safly 3 is piaoing 14044
9
safly 4 is piaoing 12996
16
主 12932
5.202786684036255
Process finished with exit code 0
使用submit异步调用
异步调用: 提交/调用一个任务,不在原地等着,直接执行下一行代码
# from multiprocessing import Process,Pool
from concurrent.futures import ProcessPoolExecutor
import time, random, os
def piao(name, n):
print('%s is piaoing %s' % (name, os.getpid()))
time.sleep(1)
return n ** 2
if __name__ == '__main__':
p = ProcessPoolExecutor(2)
objs = []
start = time.time()
for i in range(5):
obj = p.submit(piao, 'safly %s' % i, i) # 异步调用
objs.append(obj)
p.shutdown(wait=True)
print('主', os.getpid())
for obj in objs:
print(obj.result())
stop = time.time()
print(stop - start)
输出如下:
E:\python\python_sdk\python.exe "E:/python/py_pro/4 进程池.py"
safly 0 is piaoing 1548
safly 1 is piaoing 7872
safly 2 is piaoing 1548
safly 3 is piaoing 7872
safly 4 is piaoing 1548
主 7808
0
1
4
9
16
3.202626943588257
输出信息的换行是我标识有输出停顿的
简单说下执行流程:
由于进程池容量是容纳2个进程,所以会2+2+1 三次进入线程池执行,花费3秒
如果我们改下上面的代码,修改的代码如下:
from concurrent.futures import ProcessPoolExecutor
import time, random, os
def piao(name, n):
print('%s is piaoing %s' % (name, os.getpid()))
time.sleep(1)
return n ** 2
if __name__ == '__main__':
p = ProcessPoolExecutor(2)
objs = []
start = time.time()
for i in range(5):
obj = p.submit(piao, 'safly %s' % i, i) # 异步调用
objs.append(obj)
for obj in objs:
print(obj.result())
p.shutdown(wait=True)
print('主', os.getpid())
stop = time.time()
print(stop - start)
输出如下:(同样我用换行,标识出输出的时间段了)
E:\python\python_sdk\python.exe "E:/python/py_pro/4 进程池.py"
safly 0 is piaoing 7852
safly 1 is piaoing 8484
safly 2 is piaoing 7852
0
safly 3 is piaoing 8484
1
safly 4 is piaoing 7852
4
9
16
主 6816
3.178352117538452
进程池实现ftp
服务端:
from socket import *
from concurrent.futures import ProcessPoolExecutor
import os
server=socket(AF_INET,SOCK_STREAM)
server.setsockopt(SOL_SOCKET,SO_REUSEADDR,1)
server.bind(('127.0.0.1',8080))
server.listen(5)
def talk(conn,client_addr):
print('进程pid: %s' %os.getpid())
while True:
try:
msg=conn.recv(1024)
if not msg:break
conn.send(msg.upper())
except Exception:
break
if __name__ == '__main__':
p=ProcessPoolExecutor(5)
while True:
conn,client_addr=server.accept()
p.submit(talk,conn,client_addr)
客户端:
from socket import *
client=socket(AF_INET,SOCK_STREAM)
client.connect(('127.0.0.1',8080))
while True:
msg=input('>>: ').strip()
if not msg:continue
client.send(msg.encode('utf-8'))
msg=client.recv(1024)
print(msg.decode('utf-8'))