异步【ThreadPoolExecutor】和【ProcessPoolExecutor】实例

最新推荐文章于 2021-06-08 22:03:01 发布
最新推荐文章于 2021-06-08 22:03:01 发布
阅读量593 收藏 2
点赞数
原文链接：https://blog.csdn.net/qq_41922768/article/details/84134852
版权

原因：
concurrent.futures模块提供了高度封装的异步调用接口
ThreadPoolExecutor：线程池，提供异步调用
ProcessPoolExecutor: 进程池，提供异步调用

1.简单ProcessPoolExecutor: 进程池

# coding=utf-8

# 用法
import logging
from concurrent.futures import ProcessPoolExecutor
import os, time, random

logging.basicConfig(
    level=logging.INFO,
    format="%(asctime)s [*] %(processName)s  %(message)s"
)


def task (n):
    logging.info('%s is runing' % os.getpid())
    time.sleep(random.randint(1, 3))
    return n ** 2


if __name__ == '__main__':
    start = time.time()
    executor = ProcessPoolExecutor(max_workers=3)
    futures = []
    for i in range(11):
        future = executor.submit(task, i)
        futures.append(future)
    for future in futures:
        logging.info(future.result())
    executor.shutdown(True)
    logging.info(time.time() - start)

输出：

2019-10-07 20:27:25,089 [*] SpawnProcess-2  8168 is runing
2019-10-07 20:27:25,090 [*] SpawnProcess-1  4764 is runing
2019-10-07 20:27:25,095 [*] SpawnProcess-3  4368 is runing
2019-10-07 20:27:26,095 [*] SpawnProcess-3  4368 is runing
2019-10-07 20:27:27,090 [*] SpawnProcess-1  4764 is runing
2019-10-07 20:27:28,090 [*] SpawnProcess-2  8168 is runing
2019-10-07 20:27:28,091 [*] MainProcess  0
2019-10-07 20:27:28,092 [*] MainProcess  1
2019-10-07 20:27:28,092 [*] SpawnProcess-1  4764 is runing
2019-10-07 20:27:28,092 [*] MainProcess  4
2019-10-07 20:27:29,095 [*] SpawnProcess-3  4368 is runing
2019-10-07 20:27:29,095 [*] MainProcess  9
2019-10-07 20:27:29,095 [*] MainProcess  16
2019-10-07 20:27:31,094 [*] SpawnProcess-2  8168 is runing
2019-10-07 20:27:31,094 [*] SpawnProcess-1  4764 is runing
2019-10-07 20:27:31,095 [*] MainProcess  25
2019-10-07 20:27:31,096 [*] MainProcess  36
2019-10-07 20:27:31,096 [*] SpawnProcess-3  4368 is runing
2019-10-07 20:27:31,097 [*] MainProcess  49
2019-10-07 20:27:32,096 [*] MainProcess  64
2019-10-07 20:27:33,095 [*] MainProcess  81
2019-10-07 20:27:34,097 [*] MainProcess  100
2019-10-07 20:27:34,122 [*] MainProcess  9.143441438674927

2.简单ThreadPoolExecutor：线程池

# coding=utf-8

# 用法
import logging
from concurrent.futures import ThreadPoolExecutor
import os, time, random

logging.basicConfig(
    level=logging.INFO,
    format="%(asctime)s [*] %(processName)s %(threadName)s %(message)s"
)


def task (n):
    logging.info('%s is runing' % os.getpid())
    time.sleep(random.randint(1, 3))
    return n ** 2


if __name__ == '__main__':
    start = time.time()
    executor = ThreadPoolExecutor(max_workers=3)
    futures = []
    for i in range(11):
        future = executor.submit(task, i)
        futures.append(future)
    for future in futures:
        logging.info(future.result())
    executor.shutdown(True)
    logging.info(time.time() - start)

输出：

2019-10-07 20:27:53,067 [*] MainProcess ThreadPoolExecutor-0_0 5036 is runing
2019-10-07 20:27:53,067 [*] MainProcess ThreadPoolExecutor-0_1 5036 is runing
2019-10-07 20:27:53,067 [*] MainProcess ThreadPoolExecutor-0_2 5036 is runing
2019-10-07 20:27:54,084 [*] MainProcess ThreadPoolExecutor-0_2 5036 is runing
2019-10-07 20:27:55,085 [*] MainProcess ThreadPoolExecutor-0_1 5036 is runing
2019-10-07 20:27:55,085 [*] MainProcess ThreadPoolExecutor-0_2 5036 is runing
2019-10-07 20:27:56,079 [*] MainProcess ThreadPoolExecutor-0_0 5036 is runing
2019-10-07 20:27:56,079 [*] MainProcess MainThread 0
2019-10-07 20:27:56,080 [*] MainProcess MainThread 1
2019-10-07 20:27:56,080 [*] MainProcess MainThread 4
2019-10-07 20:27:56,081 [*] MainProcess MainThread 9
2019-10-07 20:27:57,087 [*] MainProcess ThreadPoolExecutor-0_1 5036 is runing
2019-10-07 20:27:57,087 [*] MainProcess MainThread 16
2019-10-07 20:27:58,080 [*] MainProcess ThreadPoolExecutor-0_0 5036 is runing
2019-10-07 20:27:58,087 [*] MainProcess ThreadPoolExecutor-0_2 5036 is runing
2019-10-07 20:27:58,088 [*] MainProcess MainThread 25
2019-10-07 20:27:58,088 [*] MainProcess MainThread 36
2019-10-07 20:27:59,089 [*] MainProcess ThreadPoolExecutor-0_2 5036 is runing
2019-10-07 20:28:00,089 [*] MainProcess MainThread 49
2019-10-07 20:28:01,081 [*] MainProcess MainThread 64
2019-10-07 20:28:01,081 [*] MainProcess MainThread 81
2019-10-07 20:28:01,090 [*] MainProcess MainThread 100
2019-10-07 20:28:01,091 [*] MainProcess MainThread 8.02418041229248

3.ProcessPoolExecutor: 进程池map的用法

# coding=utf-8

# 用法
import logging
from concurrent.futures import ProcessPoolExecutor
import os, time, random

logging.basicConfig(
    level=logging.INFO,
    format="%(asctime)s [*] %(processName)s  %(message)s"
)


def task (n):
    logging.info('%s is runing' % os.getpid())
    time.sleep(random.randint(1, 3))
    return n ** 2


if __name__ == '__main__':
    start = time.time()
    executor = ProcessPoolExecutor(max_workers=3)
    # map-------------------------------------
    futures = executor.map(task, range(1, 12))
    for future in futures:
        logging.info(future)
    
    executor.shutdown(True)
    logging.info(time.time() - start)

输出：

2019-10-07 20:33:39,853 [*] SpawnProcess-1  276 is runing
2019-10-07 20:33:39,854 [*] SpawnProcess-2  3420 is runing
2019-10-07 20:33:39,855 [*] SpawnProcess-3  9192 is runing
2019-10-07 20:33:41,855 [*] SpawnProcess-2  3420 is runing
2019-10-07 20:33:41,856 [*] SpawnProcess-3  9192 is runing
2019-10-07 20:33:42,860 [*] SpawnProcess-2  3420 is runing
2019-10-07 20:33:42,860 [*] SpawnProcess-3  9192 is runing
2019-10-07 20:33:42,860 [*] SpawnProcess-1  276 is runing
2019-10-07 20:33:42,860 [*] MainProcess  1
2019-10-07 20:33:42,860 [*] MainProcess  4
2019-10-07 20:33:42,860 [*] MainProcess  9
2019-10-07 20:33:42,860 [*] MainProcess  16
2019-10-07 20:33:42,860 [*] MainProcess  25
2019-10-07 20:33:43,862 [*] SpawnProcess-2  3420 is runing
2019-10-07 20:33:43,862 [*] SpawnProcess-1  276 is runing
2019-10-07 20:33:43,862 [*] MainProcess  36
2019-10-07 20:33:44,863 [*] SpawnProcess-2  3420 is runing
2019-10-07 20:33:45,862 [*] MainProcess  49
2019-10-07 20:33:45,863 [*] MainProcess  64
2019-10-07 20:33:45,863 [*] MainProcess  81
2019-10-07 20:33:45,865 [*] MainProcess  100
2019-10-07 20:33:46,864 [*] MainProcess  121
2019-10-07 20:33:46,890 [*] MainProcess  7.154351472854614

4.ThreadPoolExecutor：线程池 map的用法

# coding=utf-8

# 用法
import logging
from concurrent.futures import ThreadPoolExecutor
import os, time, random

logging.basicConfig(
    level=logging.INFO,
    format="%(asctime)s [*]  %(threadName)s %(message)s"
)


def task (n):
    logging.info('%s is runing' % os.getpid())
    time.sleep(random.randint(1, 3))
    return n ** 2


if __name__ == '__main__':
    start = time.time()
    executor = ThreadPoolExecutor(max_workers=3)
    # map---------------------------------------
    futures = executor.map(task, range(1, 12))
    for future in futures:
        logging.info(future)
    logging.info(time.time() - start)

输出：

2019-10-07 20:36:01,895 [*]  ThreadPoolExecutor-0_0 8228 is runing
2019-10-07 20:36:01,895 [*]  ThreadPoolExecutor-0_1 8228 is runing
2019-10-07 20:36:01,896 [*]  ThreadPoolExecutor-0_2 8228 is runing
2019-10-07 20:36:02,903 [*]  ThreadPoolExecutor-0_0 8228 is runing
2019-10-07 20:36:02,904 [*]  MainThread 1
2019-10-07 20:36:03,903 [*]  ThreadPoolExecutor-0_2 8228 is runing
2019-10-07 20:36:03,903 [*]  ThreadPoolExecutor-0_1 8228 is runing
2019-10-07 20:36:03,904 [*]  MainThread 4
2019-10-07 20:36:03,904 [*]  MainThread 9
2019-10-07 20:36:03,925 [*]  ThreadPoolExecutor-0_0 8228 is runing
2019-10-07 20:36:03,925 [*]  MainThread 16
2019-10-07 20:36:04,927 [*]  ThreadPoolExecutor-0_0 8228 is runing
2019-10-07 20:36:05,929 [*]  ThreadPoolExecutor-0_0 8228 is runing
2019-10-07 20:36:06,909 [*]  ThreadPoolExecutor-0_1 8228 is runing
2019-10-07 20:36:06,909 [*]  ThreadPoolExecutor-0_2 8228 is runing
2019-10-07 20:36:06,910 [*]  MainThread 25
2019-10-07 20:36:06,910 [*]  MainThread 36
2019-10-07 20:36:06,910 [*]  MainThread 49
2019-10-07 20:36:06,910 [*]  MainThread 64
2019-10-07 20:36:08,934 [*]  MainThread 81
2019-10-07 20:36:08,934 [*]  MainThread 100
2019-10-07 20:36:08,934 [*]  MainThread 121
2019-10-07 20:36:08,934 [*]  MainThread 7.039489984512329

5.ProcessPoolExecutor: 进程池回调函数

# coding=utf-8
import logging
from concurrent.futures import ProcessPoolExecutor
import os, time

import requests

logging.basicConfig(
    level=logging.INFO,
    format="%(asctime)s [*] %(processName)s  %(message)s"
)


def get_page (url):
    logging.info(f"<进程ID：{os.getpid()}> get：{url}")
    response = requests.get(url)
    if response.status_code == 200:
        return {'url': url, 'text': response.text}


def parse_page (res):
    res = res.result()
    url = res['url']
    size = len(res['text'])
    logging.info(f"<进程ID：{os.getpid()}> parse：{url} size：{size}")
    parse_res = f"url：<{url}>  size：<{size}>\n"
    with open('db.txt', 'a', encoding="utf-8") as f:
        f.write(parse_res)


if __name__ == '__main__':
    start = time.time()
    urls = [
        'https://www.baidu.com',
        'https://www.python.org',
        'https://www.openstack.org',
        'https://help.github.com/',
        'http://www.sina.com.cn/'
    ]
    executor = ProcessPoolExecutor(max_workers=3)
    # 异步提交任务------------------------------------------------------------
    futures = []
    for url in urls:
        future = executor.submit(get_page, url).add_done_callback(parse_page)
        futures.append(future)
    
    executor.shutdown(True)
    logging.info(time.time() - start)

输出：

2019-10-07 20:53:45,947 [*] SpawnProcess-2  <进程ID：3300> get：https://www.baidu.com
2019-10-07 20:53:45,947 [*] SpawnProcess-1  <进程ID：6116> get：https://www.python.org
2019-10-07 20:53:45,955 [*] SpawnProcess-3  <进程ID：8144> get：https://www.openstack.org
2019-10-07 20:53:46,074 [*] SpawnProcess-2  <进程ID：3300> get：https://help.github.com/
2019-10-07 20:53:46,075 [*] MainProcess  <进程ID：9156> parse：https://www.baidu.com size：2443
2019-10-07 20:53:49,089 [*] SpawnProcess-3  <进程ID：8144> get：http://www.sina.com.cn/
2019-10-07 20:53:49,089 [*] MainProcess  <进程ID：9156> parse：https://www.openstack.org size：60746
2019-10-07 20:53:49,295 [*] MainProcess  <进程ID：9156> parse：https://www.python.org size：48847
2019-10-07 20:53:49,781 [*] MainProcess  <进程ID：9156> parse：http://www.sina.com.cn/ size：547095
2019-10-07 20:53:50,436 [*] MainProcess  <进程ID：9156> parse：https://help.github.com/ size：156719
2019-10-07 20:53:50,488 [*] MainProcess  4.833276033401489

6.ThreadPoolExecutor：线程池回调函数

# coding=utf-8
import logging
from concurrent.futures import ThreadPoolExecutor
import os, time

import requests

logging.basicConfig(
    level=logging.INFO,
    format="%(asctime)s [*]  %(threadName)s %(message)s"
)


def get_page (url):
    logging.info(f"<进程ID：{os.getpid()}> get：{url}")
    response = requests.get(url)
    if response.status_code == 200:
        return {'url': url, 'text': response.text}


def parse_page (res):
    res = res.result()
    url = res['url']
    size = len(res['text'])
    logging.info(f"<进程ID：{os.getpid()}> parse：{url} size：{size}")
    parse_res = f"url：<{url}>  size：<{size}>\n"
    with open('db.txt', 'a', encoding="utf-8") as f:
        f.write(parse_res)


if __name__ == '__main__':
    start = time.time()
    urls = [
        'https://www.baidu.com',
        'https://www.python.org',
        'https://www.openstack.org',
        'https://help.github.com/',
        'http://www.sina.com.cn/'
    ]
    executor = ThreadPoolExecutor(max_workers=3)
    # 异步提交任务------------------------------------------------------------
    futures = []
    for url in urls:
        future = executor.submit(get_page, url).add_done_callback(parse_page)
        futures.append(future)
    
    executor.shutdown(True)
    logging.info(time.time() - start)

输出：

2019-10-07 20:57:20,323 [*]  ThreadPoolExecutor-0_0 <进程ID：2956> get：https://www.baidu.com
2019-10-07 20:57:20,323 [*]  ThreadPoolExecutor-0_1 <进程ID：2956> get：https://www.python.org
2019-10-07 20:57:20,324 [*]  ThreadPoolExecutor-0_2 <进程ID：2956> get：https://www.openstack.org
2019-10-07 20:57:20,460 [*]  ThreadPoolExecutor-0_0 <进程ID：2956> parse：https://www.baidu.com size：2443
2019-10-07 20:57:20,460 [*]  ThreadPoolExecutor-0_0 <进程ID：2956> get：https://help.github.com/
2019-10-07 20:57:21,169 [*]  ThreadPoolExecutor-0_2 <进程ID：2956> parse：https://www.openstack.org size：60746
2019-10-07 20:57:21,171 [*]  ThreadPoolExecutor-0_2 <进程ID：2956> get：http://www.sina.com.cn/
2019-10-07 20:57:21,587 [*]  ThreadPoolExecutor-0_2 <进程ID：2956> parse：http://www.sina.com.cn/ size：547095
2019-10-07 20:57:21,736 [*]  ThreadPoolExecutor-0_0 <进程ID：2956> parse：https://help.github.com/ size：156719
2019-10-07 20:57:23,799 [*]  ThreadPoolExecutor-0_1 <进程ID：2956> parse：https://www.python.org size：48847
2019-10-07 20:57:23,799 [*]  MainThread 3.4770350456237793
迷心兔
关注
  • 0
    点赞
  • 踩
  • 2
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
专栏目录
python 异步api ThreadPoolExecutor 、ProcessPoolExecutor(多线程、多进程)
yuhengshi的博客
04-09 626
python 异步api ThreadPoolExecutor 、ProcessPoolExecutor(多线程、多进程) python==3.7 线程 import time from concurrent.futures import ThreadPoolExecutor, as_completed from concurrent.futures._base import Future from random import randint worker_num = 5 executor = Thr
ProcessPoolExecutor小用例
luhouxiang的专栏
04-05 603
ProcessPoolExecutor用起来很简单，完全可以当成一个小模块使用，使用时通过submit函数注册需要调用的work函数，然后通过add_done_callback注册需要回调结果的函数when_done即可，当work被进程执行完成后，结果会通过回调when_done返回 demo代码如下： # !/usr/bin/env python3 # -*- coding: utf-8 -*...
[python] ThreadPoolExecutor线程池和ProcessPoolExecutor进程池
weixin_40030584的博客
11-05 5071
引言 Python标准库为我们提供了threading和multiprocessing模块编写相应的多线程/多进程代码，但是当项目达到一定的规模，频繁创建/销毁进程或者线程是非常消耗资源的，这个时候我们就要编写自己的线程池/进程池，以空间换时间。但从Python3.2开始，标准库为我们提供了concurrent.futures模块，它提供了ThreadPoolExecutor和ProcessPoo...
【Python随笔】python进程池ProcessPoolExecutor的用法与实现分析
热门推荐
HiKariのTechLab
06-08 3万+
在python开发期间，由于GIL的原因，不能直接采用并行的方式处理代码逻辑。在multiprocessing库的支持下，python程序能够启动子进程执行特定的任务，但子进程的管理也成为了问题。为了简化用户开发成本，python在concurrent.futures下内置了ProcessPoolExecutor这一数据结构，实现了简单的进程管理及任务调度。如果没有特别的需求，开发者只需要用ProcessPoolExecutor即可实现并行执行任务。因此，本文简单对ProcessPoolExecutor的实
Python标准模块--concurrent.futures模块（ThreadPoolExecutor：线程池，提供异步调用、ProcessPoolExecutor: 进程池，提供异步调用）
。
11-16 6004
目录 ProcessPoolExecutor: 进程池 ThreadPoolExecutor：线程池  map的用法  回调函数 https://docs.python.org/dev/library/concurrent.futures.html 1 介绍 concurrent.futures模块提供了高度封装的异步调用接口 ThreadPoolExecutor：线程池，提供异步调用...
java ThreadPoolExecutor 并发调用实例详解
08-30
java ThreadPoolExecutor 并发调用实例详解 java 中的并发编程是指在多个线程中执行多个任务，以提高程序的执行效率。在 Java 中，有多种方式可以实现并发编程，包括使用 Thread、Runnable、Callable、Future、...
线程池ThreadPoolExecutor使用简介与方法实例
08-26
线程池ThreadPoolExecutor使用简介与方法实例 线程池ThreadPoolExecutor是Java并发编程中...ThreadPoolExecutor是Java并发编程中一个非常重要的概念，它可以帮助我们更好地处理任务，从而提高系统的可扩展性和灵活性。
Java线程池ThreadPoolExecutor原理及使用实例
08-19
Java线程池ThreadPoolExecutor原理及使用实例 Java线程池ThreadPoolExecutor是Java并发编程中的一种基本机制，主要用于管理和执行任务的线程池。下面对其原理和使用实例进行详细介绍。 线程池概述 线程池是一个...
线程池实例：使用Executors和ThreadPoolExecutor
08-12
NULL 博文链接：https://bijian1013.iteye.com/blog/2284676
Python 异步协程函数原理及实例详解
09-18
本文将深入探讨Python异步协程函数的原理和实例。 首先，Python 3.4引入了asyncio标准库，这是一个内置的事件循环（message loop）系统，专门用于处理异步I/O操作。事件循环是异步编程的核心，它负责调度协程的执行...
python 多进程并行编程 ProcessPoolExecutor的实现
09-18
主要介绍了python 多进程并行编程 ProcessPoolExecutor的实现，文中通过示例代码介绍的非常详细，对大家的学习或者工作具有一定的参考学习价值，需要的朋友们下面随着小编来一起学习学习吧
python基础-进程池、submit同异步调用、shutdown参数、ProcessPoolExecutor进程池、进程池ftp
全干工程师
11-30 2万+
引入进程池 使用ProcessPoolExecutor进程池 使用shutdown 使用submit同步调用 使用submit异步调用引入进程池在学习线程池之前，我们先看一个例子from multiprocessing import Process import time def task(name): print("name",name) time.sleep(1)if __n
【Python测试开发】使用ProcessPoolExecutor或ThreadPoolExecutor实现多进程或多线程
刘春明的博客
08-01 4973
并发编程会给我们的程序带来极大的性能提升，并发编程具有非常广泛的应用，比如服务器、网络爬虫、性能测试等。Python中并发编程的实现方式有：多线程、多进程以及协程。本文主要介绍多线程和多进程。 1. 几个重要概念 在python并发编程之前，我们需要明确和掌握几个重要的概念，并发与并行，同步和异步，阻塞与非阻塞。 1.1 并发执行和并行执行 并行（parallel）： 指的是互不干扰的在同一时刻...
python 底层原理processpoolexecutor_python 多进程并行编程 ProcessPoolExecutor的实现
weixin_33308507的博客
02-19 331
使用 ProcessPoolExecutorfrom concurrent.futures import ProcessPoolExecutor, as_completedimport random斐波那契数列当 n 大于 30 时抛出异常def fib(n):if n > 30:raise Exception('can not > 30, now %s' % n)if n <=...
ProcessPoolExecutor
s1162276945的博客
01-15 3008
from concurrent.futures import ProcessPoolExecutor import requests import os def get_page(url): print('<%s> is getting [%s]' % (os.getpid(), url)) response = requests.get(url) if r...
concurrent.futures ProcessPoolExecutor 多进程通信
Yqiu的博客
10-18 2831
concurrent.futures ProcessPoolExecutor多进程通信 一般的多进程通信可以采用 queue,pipe,信号量等方式进行， 这里的多进程包 使用队列queue通信只能使用multiprocessing.Manages中的 Queue进行通信 例： from concurrent.futures import ProcessPoolExecutor from m...
python processpoolexecutor_Python多进程解决方案multiprocessing ProcessPoolExecutor
weixin_39599046的博客
12-10 482
大多数编程语言都会有多线程和多进程的概念，至于线程和进程的概念，大家可以百度一下。作为一门胶水语言，Python毫不意外，也可以利用多线程和多进程处理并发问题，但是多线程由于GIL的存在，起作用范围大打折扣，仅限于在IO等场景可以发挥点作用。所以，今天要跟大家分享的是Python多进程方案，更好地利用系统多核，从而提升性能。基础方案一：利用Process新建一个子进程，在子进程执行任务。我们写一个...
测试tornado的多进程性能tornado.process
LFGxiaogang的博客
08-23 2051
随后测试了下tornado的 tornado.process模块,  结果性能方面不咋地。  他的实现原理也很简单，就是主进程实例化listen fd，然后fork子进程，这样子进程继承了父进程的listen fd .  当一个新连接过来的时候，多个子进程如果都在accept，那么会出现惊群操作，当然最后只有一个进程accept拿到数据。 每个子进程有单独的epoll池里，不仅要监控listen...
ThreadPoolExecutor和ProcessPoolExecutor区别
最新发布
03-24
ThreadPoolExecutor和ProcessPoolExecutor是Python中用于并发执行任务的两个类。它们的主要区别在于执行任务的方式和适用场景。 ThreadPoolExecutor是线程池执行器，它使用线程来执行任务。线程是轻量级的执行单元...

“相关推荐”对你有帮助么？

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值