Python threading queue（不浪费内存）实例

最新推荐文章于 2024-06-20 18:33:48 发布

迷心兔

最新推荐文章于 2024-06-20 18:33:48 发布

阅读量2.1k

点赞数

原文链接：https://www.cnblogs.com/xiaobeibei26/p/6490350.html

版权

Python Threading 专栏收录该内容

10 篇文章 0 订阅

订阅专栏

原因：有需要才创建线程，且重复利用空闲线程

1.有问题的例子：

注意：这个例子是一开始就创建了所有出线程，浪费内存，线程也没有重复利用

# coding=utf-8
import queue
import time
import logging
import threading

logging.basicConfig(
    level=logging.INFO,
    format=" %(asctime)s [*] %(processName)s  %(threadName)-2s %(message)s"
)


class ThreadPool(object):
    def __init__ (self, max_num=20):  # 默认最大20个线程
        self.queue = queue.Queue(max_num)
        for i in range(max_num):  # 将20个线程类名放进队列
            self.queue.put(threading.Thread)
    
    def get_thread (self):  # 获取线程，没有的话会在这里等待
        return self.queue.get()
    
    def add_thread (self):  # 添加线程
        self.queue.put(threading.Thread)


def f1 (p, args):
    time.sleep(1)
    logging.info(args)
    p.add_thread()  # 函数执行完毕之后，自动添加线程回池


if __name__ == '__main__':
    p = ThreadPool()
    
    for i in range(100):
        ret = p.get_thread()  # 获得线程类名
        t = ret(name="A-%s" % i, target=f1, args=(p, i))  # 准备启动线程
        t.start()  # 启动线程

输出：

 2019-10-07 11:26:37,193 [*] MainProcess  A-1 1
 2019-10-07 11:26:37,193 [*] MainProcess  A-0 0
 2019-10-07 11:26:37,209 [*] MainProcess  A-19 19
 2019-10-07 11:26:37,209 [*] MainProcess  A-16 16
 2019-10-07 11:26:37,209 [*] MainProcess  A-14 14
 2019-10-07 11:26:37,209 [*] MainProcess  A-12 12
 2019-10-07 11:26:37,209 [*] MainProcess  A-10 10
 2019-10-07 11:26:37,209 [*] MainProcess  A-8 8
 2019-10-07 11:26:37,209 [*] MainProcess  A-6 6
 2019-10-07 11:26:37,209 [*] MainProcess  A-4 4
 2019-10-07 11:26:37,209 [*] MainProcess  A-2 2
 2019-10-07 11:26:37,209 [*] MainProcess  A-17 17
 2019-10-07 11:26:37,209 [*] MainProcess  A-15 15
 2019-10-07 11:26:37,209 [*] MainProcess  A-11 11
 2019-10-07 11:26:37,209 [*] MainProcess  A-7 7
 2019-10-07 11:26:37,209 [*] MainProcess  A-3 3
 2019-10-07 11:26:37,209 [*] MainProcess  A-13 13
 2019-10-07 11:26:37,209 [*] MainProcess  A-5 5
 2019-10-07 11:26:37,209 [*] MainProcess  A-9 9
 2019-10-07 11:26:37,209 [*] MainProcess  A-18 18
 2019-10-07 11:26:38,210 [*] MainProcess  A-21 21
 2019-10-07 11:26:38,210 [*] MainProcess  A-20 20
 2019-10-07 11:26:38,226 [*] MainProcess  A-25 25
 2019-10-07 11:26:38,226 [*] MainProcess  A-26 26
 2019-10-07 11:26:38,226 [*] MainProcess  A-22 22
 2019-10-07 11:26:38,226 [*] MainProcess  A-24 24
 2019-10-07 11:26:38,226 [*] MainProcess  A-23 23
 2019-10-07 11:26:38,226 [*] MainProcess  A-27 27
 2019-10-07 11:26:38,242 [*] MainProcess  A-37 37
 2019-10-07 11:26:38,242 [*] MainProcess  A-39 39
 2019-10-07 11:26:38,242 [*] MainProcess  A-36 36
 2019-10-07 11:26:38,242 [*] MainProcess  A-38 38
 2019-10-07 11:26:38,242 [*] MainProcess  A-35 35
 2019-10-07 11:26:38,242 [*] MainProcess  A-34 34
 2019-10-07 11:26:38,242 [*] MainProcess  A-33 33
 2019-10-07 11:26:38,242 [*] MainProcess  A-32 32
 2019-10-07 11:26:38,242 [*] MainProcess  A-31 31
 2019-10-07 11:26:38,242 [*] MainProcess  A-30 30
 2019-10-07 11:26:38,242 [*] MainProcess  A-29 29
 2019-10-07 11:26:38,242 [*] MainProcess  A-28 28
 2019-10-07 11:26:39,228 [*] MainProcess  A-41 41
 2019-10-07 11:26:39,228 [*] MainProcess  A-40 40
 2019-10-07 11:26:39,244 [*] MainProcess  A-45 45
 2019-10-07 11:26:39,244 [*] MainProcess  A-46 46
 2019-10-07 11:26:39,244 [*] MainProcess  A-43 43
 2019-10-07 11:26:39,244 [*] MainProcess  A-47 47
 2019-10-07 11:26:39,244 [*] MainProcess  A-44 44
 2019-10-07 11:26:39,244 [*] MainProcess  A-42 42
 2019-10-07 11:26:39,259 [*] MainProcess  A-58 58
 2019-10-07 11:26:39,259 [*] MainProcess  A-56 56
 2019-10-07 11:26:39,259 [*] MainProcess  A-53 53
 2019-10-07 11:26:39,259 [*] MainProcess  A-52 52
 2019-10-07 11:26:39,259 [*] MainProcess  A-50 50
 2019-10-07 11:26:39,259 [*] MainProcess  A-48 48
 2019-10-07 11:26:39,259 [*] MainProcess  A-57 57
 2019-10-07 11:26:39,259 [*] MainProcess  A-54 54
 2019-10-07 11:26:39,259 [*] MainProcess  A-51 51
 2019-10-07 11:26:39,259 [*] MainProcess  A-59 59
 2019-10-07 11:26:39,259 [*] MainProcess  A-55 55
 2019-10-07 11:26:39,259 [*] MainProcess  A-49 49
 2019-10-07 11:26:40,244 [*] MainProcess  A-61 61
 2019-10-07 11:26:40,244 [*] MainProcess  A-60 60
 2019-10-07 11:26:40,260 [*] MainProcess  A-64 64
 2019-10-07 11:26:40,260 [*] MainProcess  A-65 65
 2019-10-07 11:26:40,260 [*] MainProcess  A-62 62
 2019-10-07 11:26:40,260 [*] MainProcess  A-66 66
 2019-10-07 11:26:40,260 [*] MainProcess  A-67 67
 2019-10-07 11:26:40,260 [*] MainProcess  A-63 63
 2019-10-07 11:26:40,276 [*] MainProcess  A-75 75
 2019-10-07 11:26:40,276 [*] MainProcess  A-76 76
 2019-10-07 11:26:40,277 [*] MainProcess  A-74 74
 2019-10-07 11:26:40,277 [*] MainProcess  A-73 73
 2019-10-07 11:26:40,277 [*] MainProcess  A-71 71
 2019-10-07 11:26:40,277 [*] MainProcess  A-70 70
 2019-10-07 11:26:40,277 [*] MainProcess  A-72 72
 2019-10-07 11:26:40,278 [*] MainProcess  A-69 69
 2019-10-07 11:26:40,278 [*] MainProcess  A-68 68
 2019-10-07 11:26:40,298 [*] MainProcess  A-79 79
 2019-10-07 11:26:40,298 [*] MainProcess  A-78 78
 2019-10-07 11:26:40,298 [*] MainProcess  A-77 77
 2019-10-07 11:26:41,261 [*] MainProcess  A-81 81
 2019-10-07 11:26:41,261 [*] MainProcess  A-80 80
 2019-10-07 11:26:41,276 [*] MainProcess  A-85 85
 2019-10-07 11:26:41,277 [*] MainProcess  A-87 87
 2019-10-07 11:26:41,277 [*] MainProcess  A-83 83
 2019-10-07 11:26:41,277 [*] MainProcess  A-82 82
 2019-10-07 11:26:41,278 [*] MainProcess  A-86 86
 2019-10-07 11:26:41,278 [*] MainProcess  A-84 84
 2019-10-07 11:26:41,299 [*] MainProcess  A-96 96
 2019-10-07 11:26:41,299 [*] MainProcess  A-95 95
 2019-10-07 11:26:41,299 [*] MainProcess  A-93 93
 2019-10-07 11:26:41,299 [*] MainProcess  A-92 92
 2019-10-07 11:26:41,299 [*] MainProcess  A-91 91
 2019-10-07 11:26:41,299 [*] MainProcess  A-90 90
 2019-10-07 11:26:41,299 [*] MainProcess  A-88 88
 2019-10-07 11:26:41,299 [*] MainProcess  A-89 89
 2019-10-07 11:26:41,299 [*] MainProcess  A-94 94
 2019-10-07 11:26:41,314 [*] MainProcess  A-99 99
 2019-10-07 11:26:41,314 [*] MainProcess  A-97 97
 2019-10-07 11:26:41,314 [*] MainProcess  A-98 98

2.解决上面出现的问题的例子：

# coding=utf-8
import queue
import logging
import threading

logging.basicConfig(
    level=logging.INFO,
    format=" %(asctime)s [*] %(threadName)-2s %(message)s"
)

StopEvent = ()  # 暂停的事件的列表


class ThreadPool(object):
    
    def __init__ (self, max_num, min_num):
        self.free = True  # 是空闲线程
        self.q = queue.Queue()  # 创建一个队列 (func, args, callback)
        self.max_num = max_num  # 最多创建的线程数
        self.min_num = min_num  # 最多创建的线程数
        self.generate_list = []  # 真实工作创建的线程数
    
    def run (self, func, args, callback=None):
        data = (func, args, callback)  # 将任务封装进一个元组
        self.q.put(data)  # 主线程将任务放进队列
        # 如果当前的线程是u空闲的，和【工作线程的列表数量】小于线程池里的数量
        if self.free == True and len(self.generate_list) < self.max_num:
            self.work()  # 启动线程工作
    
    def work (self):
        for _ in range(self.min_num):
            thread = threading.Thread(target=self.call)
            thread.start()
    
    def call (self):
        # 获取当前线程的对象，就一个线程对象
        current_thread = threading.currentThread
        # 把当前线程的对象添加到工作线程的列表中
        self.generate_list.append(current_thread)
        
        # 获取队列里的对像，并执行func函数
        job = self.q.get()
        # 当队列里的数据不为空时，就取出数据
        while job != StopEvent:
            func, args, callback = job
            try:  # 执行任务
                func(args)
            except Exception as e:
                print(e)
            # 不是空闲的线程
            self.free = False
            # 从队列中获取新的任务
            job = self.q.get()
        
        else:
            self.generate_list.remove(current_thread)  # 清除该线程，让Python垃圾回收机制处理
    
    def close (self):
        for _ in range(len(self.generate_list)):
            self.q.put(StopEvent)


def func (args):
    # time.sleep(1)  # 如果要等待，就用最大的线程数执行，五秒结束
    logging.info(f"参数：{args}")


if __name__ == '__main__':
    pool = ThreadPool(10, 2)  # 创建10个线程的线程池
    for i in range(1, 51):  # 执行func函数50次
        pool.run(func=func, args=(i,))
    
    logging.info(f"线程的数量：{len(pool.generate_list)}")
    pool.close()

输出：

 2019-10-07 13:15:47,795 [*] Thread-1 参数：(1,)
 2019-10-07 13:15:47,795 [*] MainThread 线程的数量：2
 2019-10-07 13:15:47,795 [*] Thread-1 参数：(2,)
 2019-10-07 13:15:47,795 [*] Thread-1 参数：(3,)
 2019-10-07 13:15:47,795 [*] Thread-1 参数：(4,)
 2019-10-07 13:15:47,795 [*] Thread-1 参数：(5,)
 2019-10-07 13:15:47,795 [*] Thread-1 参数：(6,)
 2019-10-07 13:15:47,795 [*] Thread-1 参数：(7,)
 2019-10-07 13:15:47,795 [*] Thread-1 参数：(8,)
 2019-10-07 13:15:47,795 [*] Thread-1 参数：(9,)
 2019-10-07 13:15:47,795 [*] Thread-1 参数：(10,)
 2019-10-07 13:15:47,795 [*] Thread-1 参数：(11,)
 2019-10-07 13:15:47,795 [*] Thread-1 参数：(12,)
 2019-10-07 13:15:47,795 [*] Thread-1 参数：(13,)
 2019-10-07 13:15:47,795 [*] Thread-1 参数：(14,)
 2019-10-07 13:15:47,795 [*] Thread-1 参数：(15,)
 2019-10-07 13:15:47,795 [*] Thread-1 参数：(16,)
 2019-10-07 13:15:47,795 [*] Thread-2 参数：(17,)
 2019-10-07 13:15:47,795 [*] Thread-1 参数：(18,)
 2019-10-07 13:15:47,795 [*] Thread-2 参数：(19,)
 2019-10-07 13:15:47,795 [*] Thread-1 参数：(20,)
 2019-10-07 13:15:47,795 [*] Thread-2 参数：(21,)
 2019-10-07 13:15:47,795 [*] Thread-1 参数：(22,)
 2019-10-07 13:15:47,795 [*] Thread-2 参数：(23,)
 2019-10-07 13:15:47,795 [*] Thread-1 参数：(24,)
 2019-10-07 13:15:47,795 [*] Thread-2 参数：(25,)
 2019-10-07 13:15:47,795 [*] Thread-1 参数：(26,)
 2019-10-07 13:15:47,795 [*] Thread-2 参数：(27,)
 2019-10-07 13:15:47,795 [*] Thread-1 参数：(28,)
 2019-10-07 13:15:47,795 [*] Thread-2 参数：(29,)
 2019-10-07 13:15:47,795 [*] Thread-1 参数：(30,)
 2019-10-07 13:15:47,795 [*] Thread-2 参数：(31,)
 2019-10-07 13:15:47,795 [*] Thread-1 参数：(32,)
 2019-10-07 13:15:47,795 [*] Thread-2 参数：(33,)
 2019-10-07 13:15:47,795 [*] Thread-1 参数：(34,)
 2019-10-07 13:15:47,795 [*] Thread-2 参数：(35,)
 2019-10-07 13:15:47,795 [*] Thread-1 参数：(36,)
 2019-10-07 13:15:47,795 [*] Thread-2 参数：(37,)
 2019-10-07 13:15:47,795 [*] Thread-1 参数：(38,)
 2019-10-07 13:15:47,795 [*] Thread-2 参数：(39,)
 2019-10-07 13:15:47,795 [*] Thread-1 参数：(40,)
 2019-10-07 13:15:47,795 [*] Thread-2 参数：(41,)
 2019-10-07 13:15:47,795 [*] Thread-1 参数：(42,)
 2019-10-07 13:15:47,795 [*] Thread-2 参数：(43,)
 2019-10-07 13:15:47,795 [*] Thread-1 参数：(44,)
 2019-10-07 13:15:47,795 [*] Thread-2 参数：(45,)
 2019-10-07 13:15:47,795 [*] Thread-1 参数：(46,)
 2019-10-07 13:15:47,795 [*] Thread-2 参数：(47,)
 2019-10-07 13:15:47,795 [*] Thread-1 参数：(48,)
 2019-10-07 13:15:47,795 [*] Thread-2 参数：(49,)
 2019-10-07 13:15:47,795 [*] Thread-1 参数：(50,)

迷心兔

关注

0
点赞
踩
8

收藏

觉得还不错? 一键收藏
0
评论
Python threading queue（不浪费内存）实例

原因：有需要才创建线程池，且重复利用空闲线程1.有问题的例子：注意：这个例子是一开始就创建了所有出线程，浪费内存，线程也没有重复利用# coding=utf-8import queueimport timeimport loggingimport threadinglogging.basicConfig( level=logging.INFO, format="...
复制链接

扫一扫