Python3 下分布式进程的简单应用

昨天了解了Python3 下分布式进程的概念，所以就改写了廖雪峰Python3教程中的分布式进程中的例子，只能看到分布式计算的现象，并没有考虑程序的健壮性和ketP可扩展性以及pythonic.高手请勿喷,希望有兴趣的人能改写下

Manager.py

#coding:utf-8
'''Author:Frank.ZhangLongqi
   Date:2017-3-18
   Environment:win8.1,Python3.6
   Improved:Robustness and extensibility
   Describe:重现Manager/Workers
   分布式计算模型，有两个脚本，一个是Manager.py，用于任务的存储、分配，另一个是Worker.py，用于从调度机(Manager.py)获取任务并执行，并将结果写入调度机
   本次可同时运行3个Worker机
'''
import random, time
from multiprocessing import Queue as Queue #此处要用高级队列，而非导入普通队列import Queue
from multiprocessing.managers import BaseManager
from multiprocessing import freeze_support


task_queue =  Queue()  
result_queue = Queue()
count_queue = Queue()


class QueueManager(BaseManager):  # 从BaseManager继承的QueueManager:
    pass


def return_task_queue():
    global task_queue
    return task_queue  # 返回发送任务队列
def return_result_queue ():
    global result_queue
    return result_queue # 返回接收结果队列
def return_count_queue():
    global count_queue
    return count_queue  # 返回发送任务队列


def test(counts,workers):
    QueueManager.register('get_task_queue', callable=return_task_queue)   
    QueueManager.register('get_result_queue', callable=return_result_queue)
    QueueManager.register('get_count_queue', callable=return_count_queue)


    manager = QueueManager(address=('127.0.0.1', 7000), authkey=b'abc') # 绑定端口5000, 设置验证码'abc':
    manager.start()  # 启动Queue: 
   
    task = manager.get_task_queue()   # 获得通过网络访问的Queue对象
    result = manager.get_result_queue()
    count = manager.get_count_queue()


    for _ in range(workers):    #将平均分配的任务量写入队列，传送给Worker机
        count.put(counts//workers)
    for i in range(counts):     #将任务写入任务队列
        n = random.randint(0, 1000)
        print('Put task %d...' % n)
        task.put(n)


    print('Try get results...') #在结果队列获取结果
    for i in range(counts):
        try:
            r = result.get(timeout=5)
            print('Result: %s' % r)
        except Queue.Empty:
             print('result queue is empty.')
         
    manager.shutdown()   # 关闭连接
    print('master exit.')
    
if __name__=='__main__':
    start=time.clock()
    freeze_support()
    print('start!')
    counts=90           #任务队列等待被处理的作业数
    workers=3           #Worker机器的数量
    test(counts,workers)
    end=time.clock()
    print("time internal:",end-start)

Worker.py

#coding=utf-8
'''Worker机程序 '''
import random, time
from multiprocessing import Queue as Queue
from multiprocessing.managers import BaseManager
 
class QueueManager(BaseManager):
    pass
 
def init():
    global task
    global result
    QueueManager.register('get_task_queue')  
    QueueManager.register('get_result_queue')
    QueueManager.register('get_count_queue')

    server_addr = '127.0.0.1'       # 连接到Manager，也就是分配任务的机器:
    print('Connect to server %s...' % server_addr)

    m = QueueManager(address=(server_addr, 7000), authkey=b'abc') # 输入约定好的端口和验证码
    m.connect()# 从网络连接:

    task = m.get_task_queue()# 获取Queue的对象
    result = m.get_result_queue()
    count = m.get_count_queue()
    return count.get(timeout=2)
def get_result(N):# 从task队列取任务,并把结果写入result队列
    for _ in range(N):
        try:
            n = task.get(timeout=1)
            print('run task %d * %d...' % (n, n))
            r = '%d * %d = %d' % (n, n, n*n)
            time.sleep(1)
            result.put(r)
        except Queue.Empty:
            print('task queue is empty.')
    print('worker exit.')
    
if __name__=='__main__':
    N=init()
    get_result(N)

Demon:

Manager:

D:\PYworkbench\py\distributed computing>python Manager.py
start!
Put task 82...
Put task 154...
Put task 657...
Put task 332...
Put task 267...
Put task 663...
Put task 841...
Put task 852...
Put task 552...
Put task 435...
Put task 573...
Put task 413...
Put task 580...
Put task 77...
Put task 232...
Put task 470...
Put task 684...
Put task 52...
Put task 991...
Put task 600...
Put task 287...
Put task 175...
Put task 858...
Put task 191...
Put task 151...
Put task 247...
Put task 530...
Put task 200...
Put task 466...
Put task 721...
Put task 759...
Put task 788...
Put task 703...
Put task 291...
Put task 637...
Put task 650...
Put task 325...
Put task 122...
Put task 282...
Put task 293...
Put task 354...
Put task 929...
Put task 169...
Put task 865...
Put task 850...
Put task 788...
Put task 68...
Put task 477...
Put task 965...
Put task 695...
Put task 26...
Put task 624...
Put task 873...
Put task 205...
Put task 949...
Put task 674...
Put task 48...
Put task 984...
Put task 750...
Put task 860...
Put task 710...
Put task 167...
Put task 637...
Put task 627...
Put task 138...
Put task 693...
Put task 441...
Put task 728...
Put task 196...
Put task 419...
Put task 779...
Put task 156...
Put task 418...
Put task 557...
Put task 478...
Put task 970...
Put task 220...
Put task 156...
Put task 781...
Put task 477...
Put task 43...
Put task 340...
Put task 241...
Put task 92...
Put task 232...
Put task 498...
Put task 303...
Put task 119...
Put task 982...
Put task 292...
Try get results...
Result: 82 * 82 = 6724
Result: 154 * 154 = 23716
Result: 657 * 657 = 431649
Result: 332 * 332 = 110224
Result: 267 * 267 = 71289
Result: 663 * 663 = 439569
Result: 841 * 841 = 707281
Result: 852 * 852 = 725904
Result: 552 * 552 = 304704
Result: 435 * 435 = 189225
Result: 573 * 573 = 328329
Result: 413 * 413 = 170569
Result: 580 * 580 = 336400
Result: 77 * 77 = 5929
Result: 232 * 232 = 53824
Result: 470 * 470 = 220900
Result: 684 * 684 = 467856
Result: 52 * 52 = 2704
Result: 991 * 991 = 982081
Result: 600 * 600 = 360000
Result: 287 * 287 = 82369
Result: 175 * 175 = 30625
Result: 858 * 858 = 736164
Result: 191 * 191 = 36481
Result: 151 * 151 = 22801
Result: 247 * 247 = 61009
Result: 530 * 530 = 280900
Result: 200 * 200 = 40000
Result: 466 * 466 = 217156
Result: 721 * 721 = 519841
Result: 759 * 759 = 576081
Result: 788 * 788 = 620944
Result: 703 * 703 = 494209
Result: 291 * 291 = 84681
Result: 637 * 637 = 405769
Result: 650 * 650 = 422500
Result: 325 * 325 = 105625
Result: 122 * 122 = 14884
Result: 282 * 282 = 79524
Result: 293 * 293 = 85849
Result: 354 * 354 = 125316
Result: 929 * 929 = 863041
Result: 169 * 169 = 28561
Result: 865 * 865 = 748225
Result: 850 * 850 = 722500
Result: 788 * 788 = 620944
Result: 68 * 68 = 4624
Result: 477 * 477 = 227529
Result: 965 * 965 = 931225
Result: 695 * 695 = 483025
Result: 26 * 26 = 676
Result: 624 * 624 = 389376
Result: 873 * 873 = 762129
Result: 205 * 205 = 42025
Result: 949 * 949 = 900601
Result: 674 * 674 = 454276
Result: 48 * 48 = 2304
Result: 984 * 984 = 968256
Result: 750 * 750 = 562500
Result: 860 * 860 = 739600
Result: 710 * 710 = 504100
Result: 167 * 167 = 27889
Result: 637 * 637 = 405769
Result: 627 * 627 = 393129
Result: 138 * 138 = 19044
Result: 693 * 693 = 480249
Result: 441 * 441 = 194481
Result: 728 * 728 = 529984
Result: 196 * 196 = 38416
Result: 419 * 419 = 175561
Result: 779 * 779 = 606841
Result: 156 * 156 = 24336
Result: 418 * 418 = 174724
Result: 557 * 557 = 310249
Result: 478 * 478 = 228484
Result: 970 * 970 = 940900
Result: 220 * 220 = 48400
Result: 156 * 156 = 24336
Result: 781 * 781 = 609961
Result: 477 * 477 = 227529
Result: 43 * 43 = 1849
Result: 340 * 340 = 115600
Result: 241 * 241 = 58081
Result: 92 * 92 = 8464
Result: 232 * 232 = 53824
Result: 498 * 498 = 248004
Result: 303 * 303 = 91809
Result: 119 * 119 = 14161
Result: 982 * 982 = 964324
Result: 292 * 292 = 85264
master exit.

time internal: 46.58379018408858                  

Three Workers:(同时运行的3个Worker.py)