简单的生产者/消费者实现。在
虽然来回移动数据确实会导致开销并增加内存使用,但只要多个进程不需要相同的数据,则开销最小。尝试一下,发现:)可以通过更改队列和池大小来调整内存占用。在
线程是另一种减少内存使用的方法,但代价是在GIL上被阻塞,并且如果处理是python字节码,则是有效的单线程处理。在import multiprocessing
# Some fixed size to avoid run away memory use
recorded_data = multiprocessing.Queue(100)
def process(recorded_data):
while True:
data = recorded_data.get()
def record(recorded_data):
for data in input_stream:
recorded_data.put(data)
producer = multiprocessing.Process(target=record,
args=(recorded_data,))
producer.start()
# Pool of 10 processes
num_proc = 10
consumer_pool = multiprocessing.Pool(num_proc)
results = []
for _ in xrange(num_proc):
results.append(
consumer_pool.apply_async(process,
args=(recorded_data,)))
producer.join()
# If processing actually returns something
for result in results:
print result
# Consumers wait for data from queue forever
# so terminate them when done
consumer_pool.terminate()