python学习记录一：关于分布式进程执行报错以及解决方案

vivian7fan

于 2022-06-28 17:21:10 发布

阅读量593

点赞数

分类专栏： python 文章标签： python 学习开发语言

本文链接：https://blog.csdn.net/vivian7fan/article/details/125505488

版权

python 专栏收录该内容

5 篇文章 0 订阅

订阅专栏

源代码内容如下：（来自廖雪峰官方网站-python教程-进程和线程-分布式进程）

# test_07.py
import random, queue
from multiprocessing.managers import BaseManager

# 发送任务的队列：
task_queue = queue.Queue()
# 接收结果的队列
result_queue = queue.Queue()

# 从BaseManager继承的QueueManager：
class QueueManager(BaseManager):
    pass

# 把两个Queue都注册到网络上，callable参数关联了Queue对象
QueueManager.register('get_task_queue', callable=lambda: task_queue)
QueueManager.register('get_result_queue', callable=lambda: result_queue)
# 绑定端口5000，设置验证码‘abc’
manager = QueueManager(address=('', 5000), authkey=b'abc')
# 启动Queue：
manager.start()
# 获得通过网络访问的Queue对象
task = manager.get_task_queue()
result = manager.get_result_queue()
# 放几个任务进去：
for i in range(10):
n = random.randint(0, 10000)
print('Put task %d...' % n)
task.put(n)
# 从result队列读取结果：
print('Try get result...')
for i in range(10):
r = result.get(timeout=10)
print('Result: %s' % r)
# 关闭
manager.shutdown()
print('master exit.')

报错（一）：
执行报错“_pickle.PicklingError: Can’t pickle <function at 0x0000021EED298438>: attribute lookup on main failed”，具体报错内容如下：

"D:\Program Files\python-3.7.9\python.exe" D:/py/test-three/test_07.py
Traceback (most recent call last):
  File "D:/py/test-three/test_07.py", line 22, in <module>
    manager.start()
  File "D:\Program Files\python-3.7.9\lib\multiprocessing\managers.py", line 563, in start
    self._process.start()
  File "D:\Program Files\python-3.7.9\lib\multiprocessing\process.py", line 112, in start
    self._popen = self._Popen(self)
  File "D:\Program Files\python-3.7.9\lib\multiprocessing\context.py", line 322, in _Popen
    return Popen(process_obj)
  File "D:\Program Files\python-3.7.9\lib\multiprocessing\popen_spawn_win32.py", line 89, in __init__
    reduction.dump(process_obj, to_child)
  File "D:\Program Files\python-3.7.9\lib\multiprocessing\reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
_pickle.PicklingError: Can't pickle <function <lambda> at 0x0000021EED298438>: attribute lookup <lambda> on __main__ failed

首先查看报错内容，寻找错误源头‘manager.start()’，无法启动Queue，根据“_pickle.PicklingError: Can’t pickle <function at 0x0000021EED298438>: attribute lookup on main failed”，lambda表达式无法序列化，参考文章【https://blog.csdn.net/tpc4289/article/details/79280659/】，自定义函数，修改部分代码如下：

# 自定义函数return_tast_queue
def return_task_queue():
    global task_queue
    return task_queue

# 自定义函数return_result_queue
def return_result_queue():
    global result_queue
    return result_queue

# 把两个Queue都注册到网络上，callable参数关联了Queue对象
QueueManager.register('get_task_queue', callable=return_task_queue)
QueueManager.register('get_result_queue', callable=return_result_queue)

执行修改后的代码，再次报错，报错内容变化，说明上一个问题解决，新增其他问题。
报错（二）：
is not going to be frozen to produce an executable.‘’')……具体报错内容为：

"D:\Program Files\python-3.7.9\python.exe" D:/py/test-three/test_07.py
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "D:\Program Files\python-3.7.9\lib\multiprocessing\spawn.py", line 105, in spawn_main
    exitcode = _main(fd)
  File "D:\Program Files\python-3.7.9\lib\multiprocessing\spawn.py", line 114, in _main
    prepare(preparation_data)
  File "D:\Program Files\python-3.7.9\lib\multiprocessing\spawn.py", line 225, in prepare
    _fixup_main_from_path(data['init_main_from_path'])
  File "D:\Program Files\python-3.7.9\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path
    run_name="__mp_main__")
  File "D:\Program Files\python-3.7.9\lib\runpy.py", line 263, in run_path
    pkg_name=pkg_name, script_name=fname)
  File "D:\Program Files\python-3.7.9\lib\runpy.py", line 96, in _run_module_code
    mod_name, mod_spec, pkg_name, script_name)
  File "D:\Program Files\python-3.7.9\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "D:\py\test-three\test_07.py", line 32, in <module>
    manager.start()
  File "D:\Program Files\python-3.7.9\lib\multiprocessing\managers.py", line 563, in start
    self._process.start()
  File "D:\Program Files\python-3.7.9\lib\multiprocessing\process.py", line 112, in start
    self._popen = self._Popen(self)
  File "D:\Program Files\python-3.7.9\lib\multiprocessing\context.py", line 322, in _Popen
    return Popen(process_obj)
  File "D:\Program Files\python-3.7.9\lib\multiprocessing\popen_spawn_win32.py", line 46, in __init__
    prep_data = spawn.get_preparation_data(process_obj._name)
  File "D:\Program Files\python-3.7.9\lib\multiprocessing\spawn.py", line 143, in get_preparation_data
    _check_not_importing_main()
  File "D:\Program Files\python-3.7.9\lib\multiprocessing\spawn.py", line 136, in _check_not_importing_main
    is not going to be frozen to produce an executable.''')
RuntimeError: 
        An attempt has been made to start a new process before the
        current process has finished its bootstrapping phase.

        This probably means that you are not using fork to start your
        child processes and you have forgotten to use the proper idiom
        in the main module:

            if __name__ == '__main__':
                freeze_support()
                ...

        The "freeze_support()" line can be omitted if the program
        is not going to be frozen to produce an executable.

这个报错内容很明显，需要添加“if name == ‘main’:”，具体原理可参考文章【https://blog.csdn.net/shenfuli/article/details/103969964】

报错（三）：

"D:\Program Files\python-3.7.9\python.exe" D:/py/test-three/test_08.py
Traceback (most recent call last):
  File "D:/py/test-three/test_08.py", line 36, in <module>
    task = manager.get_task_queue()
  File "D:\Program Files\python-3.7.9\lib\multiprocessing\managers.py", line 724, in temp
    token, exp = self._create(typeid, *args, **kwds)
  File "D:\Program Files\python-3.7.9\lib\multiprocessing\managers.py", line 607, in _create
    conn = self._Client(self._address, authkey=self._authkey)
  File "D:\Program Files\python-3.7.9\lib\multiprocessing\connection.py", line 492, in Client
    c = SocketClient(address)
  File "D:\Program Files\python-3.7.9\lib\multiprocessing\connection.py", line 620, in SocketClient
    s.connect(address)
OSError: [WinError 10049] 在其上下文中，该请求的地址无效。

进程已结束,退出代码1

此处就是照着源代码输入时，丢失ip，需要填入自己的IP，cmd输入‘ipconfig’查看本机ip，填入此段代码：manager = QueueManager(address=(‘此处填入本机IP’, 5000), authkey=b’abc’)

修改后的正确代码如下：

import random, queue
from multiprocessing.managers import BaseManager

# 发送任务的队列：
task_queue = queue.Queue()
# 接收结果的队列
result_queue = queue.Queue()


def return_task_queue():
    global task_queue
    return task_queue


def return_result_queue():
    global result_queue
    return result_queue


# 从BaseManager继承的QueueManager：
class QueueManager(BaseManager):
    pass


if __name__ == '__main__':
    # 把两个Queue都注册到网络上，callable参数关联了Queue对象
    QueueManager.register('get_task_queue', callable=return_task_queue)
    QueueManager.register('get_result_queue', callable=return_result_queue)
    # 绑定端口5000，设置验证码‘abc’
    manager = QueueManager(address=('本机ip', 5000), authkey=b'abc')
    # 启动Queue：
    manager.start()
    # 获得通过网络访问的Queue对象
    task = manager.get_task_queue()
    result = manager.get_result_queue()
    # 放几个任务进去：
    for i in range(10):
        n = random.randint(0, 10000)
        print('Put task %d...' % n)
        task.put(n)
    # 从result队列读取结果：
    print('Try get result...')
    for i in range(10):
        r = result.get(timeout=10)
        print('Result: %s' % r)
    # 关闭
    manager.shutdown()
    print('master exit.')