利用Queue和managers实现分布式进程

multiprocessing模块中的managers子模块支持把多进程分布到多台机器上。
首先编写服务进程，服务进程负责把任务写入task_queue, 并接收工作进程返回的result_queue
serverManager.py:

from multiprocessing.managers import BaseManager，Queue

# 1. 建立task_queue和result_queue, 用来存放任务和结果
task_queue = Queue()
result_queue = Queue()


class QueueManager(BaseManager):
    pass

# 2. 把创建的队列注册在网络上，利用register方法， callable 参数关联类Queue对象
QueueManager.register('get_task_queue', callable=lambda: task_queue)
QueueManager.register('get_result_queue', callable=lambda: result_queue)

# 3.绑定端口8000, 验证口令'123'
manager = QueueManager(address=('', 8000), authkey='123'.encode('utf-8'))
manager.start()

# 4. 通过manager获取网络注册的Queue对象
task = manager.get_task_queue()
result = manager.get_result_queue()

# 5.添加任务
for url in ["imageUrl_" + str(i) for i in range(10)]:
    print('put task %s...' % url)
    task.put(url)

# 6.获取返回结果
print('get result...')

for i in range(10):
    print('result is %s' % result.get(timeout=10))

manager.shutdown()

工作进程通过网络获取任务队列task_queue, 对任务队列中的数据进行处理后写入result_queue

workerManager.py:

import time
from multiprocessing.managers import BaseManager


class QueueManager(BaseManager):
    pass

# 1.注册获取queue
QueueManager.register('get_task_queue')
QueueManager.register('get_result_queue')

# 2.连接服务器
server_addr = '127.0.0.1'
print('connect to server')
mng = QueueManager(address=(server_addr, 8000), authkey='123'.encode('utf-8'))

# 连接
mng.connect()
# 3. 获取queue对象
task = mng.get_task_queue()
result = mng.get_result_queue()

# 4.从task获取任务，并把结果写入result
while not task.empty():
    image_url = task.get(True, timeout=5)
    print('run task download %s ...' % image_url)
    time.sleep(1)
    result.put('%s--->success' % image_url)

print('worker finish!')

首先运行服务进程，得到结果：

put task imageUrl_0...
put task imageUrl_1...
put task imageUrl_2...
put task imageUrl_3...
put task imageUrl_4...
put task imageUrl_5...
put task imageUrl_6...
put task imageUrl_7...
put task imageUrl_8...
put task imageUrl_9...
get result...

然后运行工作进程

connect to server
run task download imageUrl_0 ...
run task download imageUrl_1 ...
run task download imageUrl_2 ...
run task download imageUrl_3 ...
run task download imageUrl_4 ...
run task download imageUrl_5 ...
run task download imageUrl_6 ...
run task download imageUrl_7 ...
run task download imageUrl_8 ...
run task download imageUrl_9 ...
worker finish!

最后服务进程输出

result is imageUrl_0--->success
result is imageUrl_1--->success
result is imageUrl_2--->success
result is imageUrl_3--->success
result is imageUrl_4--->success
result is imageUrl_5--->success
result is imageUrl_6--->success
result is imageUrl_7--->success
result is imageUrl_8--->success
result is imageUrl_9--->success

转载于:https://my.oschina.net/acutesun/blog/1483859