python进程数上限_python如何控制进程或者线程的个数

背景

日常开发中，难免遇到并发场景，而并发场景难免需要做流量控制，即需要对并发的进程或者线程的总量进行控制。 今天简单总结两种常用的控制线程个数的方法。

方法一：进程池/线程池

如下例demo所示， 创建了一个大小是4的进程池，然后创建5个进程，并启动

from multiprocessing import Pool

import os, time, random

def long_time_task(name):

print('Run task %s (%s)...' % (name, os.getpid()))

start = time.time()

time.sleep(random.random() * 3)

end = time.time()

print('Task %s runs %0.2f seconds.' % (name, (end - start)))

if __name__ == '__main__':

print('Parent process %s.' % os.getpid())

p = Pool(4)

for i in range(5):

p.apply_async(long_time_task, args=(i,))

print('Waiting for all subprocesses done...')

p.close()

p.join()

print('All subprocesses done.')

运行结果如下，可以看到第5个进程会等池子里的进程完成一个后才会被启动

Run task 0 (32952)...

Run task 1 (32951)...

Run task 2 (32953)...

Run task 3 (32954)...

Task 2 runs 0.68 seconds.

Run task 4 (32953)...

Task 1 runs 1.41 seconds.

Task 0 runs 1.44 seconds.

Task 4 runs 2.15 seconds.

Task 3 runs 2.98 seconds.

All subprocesses done.

方法二：queue

queue 模块即队列，特别适合处理信息在多个线程间安全交换的多线程程序中。 下面的demo展示了如何通过queue来限制线程的并发个数

import threading

import queue

import time

import random

import os

maxThreads = 4

class Store(threading.Thread):

def __init__(self, q):

threading.Thread.__init__(self)

self.queue = q

# self.store = store

def run(self):

try:

print('Run task (%s)...' % (os.getpid()))

start = time.time()

time.sleep(random.random() * 3)

end = time.time()

t = threading.currentThread()

# 线程ID

print('Thread id : %d' % t.ident)

print('Thread name : %s' % t.getName())

print('Task runs %0.2f seconds.' % (end - start))

except Exception as e:

print(e)

finally:

self.queue.get()

self.queue.task_done()

def main():

q = queue.Queue(maxThreads)

for s in range(6):

q.put(s)

t = Store(q)

t.start()

q.join()

print('over')

if __name__ == '__main__':

main()

运行结果如下：

Run task (33259)...

Run task (33259)...

Run task (33259)...

Run task (33259)...

Thread id : 123145444999168

Thread name : Thread-13

Task runs 0.04 seconds.

Run task (33259)...

Thread id : 123145394630656

Thread name : Thread-10

Task runs 1.02 seconds.

Run task (33259)...

Thread id : 123145428209664

Thread name : Thread-12

Task runs 1.20 seconds.

Thread id : 123145394630656

Thread name : Thread-17

Task runs 0.68 seconds.

Thread id : 123145444999168

Thread name : Thread-14

Task runs 1.79 seconds.

Thread id : 123145411420160

Thread name : Thread-11

Task runs 2.96 seconds.

over

以上就是python如何控制进程或者线程的个数的详细内容，更多关于python 控制进程或线程的资料请关注我们其它相关文章！

本文标题: python如何控制进程或者线程的个数

本文地址: http://www.cppcns.com/jiaoben/python/354832.html