python并发编程笔记6

python并发编程笔记6

是根据蚂蚁学Python的视频做的笔记，方便自己后续回顾

视频链接：BV1bK411A7tV

老师的源码

这一份笔记对应的是视频的P9-P10

P9-使用多进程multiprocessing模块加速程序的运行

1、有了多线程threading，为什么还要用多进程multiprocessing

multiprocessing模块就是python为了解决GIL缺陷引入的一个模块，原理是用多进程在多CPU上并行执行，此时不是多线程，现在是多进程

（系统运行着多个python解释器进程，真正的并行计算）

2、多进程multiprocessing知识梳理

语法条目	多线程	多进程
引入模块	from threading import Thread	from multiprocessing import Process
新建	t=Thread(target=func, args=(100, ))	p = Process(target=f, args=(‘bob’,))
启动	t.start()	p.start()
等待结束	t.join()	p.join()
数据通信	import queue
q = queue.Queue()
q.put(item)
item = q.get()	from multiprocessing import Queue
q = Queue()
q.put([42, None, ‘hello’])
item = q.get()
线程安全加锁	from threading import Lock
lock = Lock()
with lock:

# do something | from multiprocessing import Lock

lock = Lock()
with lock:
# do something |
| 池化技术 | from concurrent.futures import ThreadPoolExecutor

with ThreadPoolExecutor() as pool:
# 方法1
results = pool.map(func, [1,2,3])

# 方法2
future = pool.submit(func, 1)
result = future.result() | from concurrent.futures import ProcessPoolExecutor

with ProcessPoolExecutor() as pool:
# 方法1
results = pool.map(func, [1,2,3])

# 方法2
future = pool.submit(func, 1)
result = future.result() |

注意：线程里q.put()是一项一项加 而进程里是可以一个列表一起加

3、代码实战:单线程、多线程、多进程对比CPU密集计算速度

结果：

代码：

import math
from concurrent.futures import ThreadPoolExecutor ,ProcessPoolExecutor
import time

PRIMES = [112272535095293] * 100


def is_prime(n):
    if n < 2:
        return False
    if n == 2:
        return True
    if n % 2 == 0:
        return False
    sqrt_n = int(math.floor(math.sqrt(n)))
    for i in range(3, sqrt_n + 1, 2):
        if n % i == 0:
            return False
    return True


def single_thread():
    for number in PRIMES:
        is_prime(number)


def multi_thread():
    with ThreadPoolExecutor() as pool:
        results = pool.map(is_prime,PRIMES)


def multi_process():
    with ProcessPoolExecutor() as pool:
        results = pool.map(is_prime,PRIMES)


if __name__ == '__main__':
    start = time.time()
    single_thread()
    end = time.time()
    print("single_thread,cost",end-start,"秒")

    start = time.time()
    multi_thread()
    end = time.time()
    print("multi_thread,cost", end - start, "秒")

    start = time.time()
    multi_process()
    end = time.time()
    print("multi_process,cost", end - start, "秒")

P10-Python在Flask服务中使用多进程池加速程序运行

代码：

import flask
import math
import json
from concurrent.futures import ProcessPoolExecutor

app = flask.Flask(__name__)


def is_prime(n):
    if n < 2:
        return False
    if n == 2:
        return True
    if n % 2 == 0:
        return False
    sqrt_n = int(math.floor(math.sqrt(n)))
    for i in range(3, sqrt_n + 1, 2):
        if n % i == 0:
            return False
    return True


@app.route("/is-prime/<numbers>")
def api_is_prime(numbers):
    numbers_list = [int(x) for x in numbers.split(",")]
    results = process_pool.map(is_prime, numbers_list)
    return json.dumps(dict(zip(numbers_list, results)))


if __name__ == '__main__':
    process_pool = ProcessPoolExecutor()
    app.run()

注意：

多进程的代码尽管与多线程的极为相同，但进程池必须放到if name == ‘main’:中。

因为多进程的环境之间都是互相完全隔离的，定义pool时，他所依赖的函数都必须声明完了。

不像多线程定义在哪里都可以，共享当前进程的多有环境。

大部分情况下使用多线程就好了。