python学习记录一：关于分布式进程执行报错以及解决方案

57 阅读 0 评论 38 点赞

我是靠谱客的博主贪玩小笼包，最近开发中收集的这篇文章主要介绍python学习记录一：关于分布式进程执行报错以及解决方案，觉得挺不错的，现在分享给大家，希望可以做个参考。

概述

源代码内容如下：（来自廖雪峰官方网站-python教程-进程和线程-分布式进程）

# test_07.py
import random, queue
from multiprocessing.managers import BaseManager
# 发送任务的队列：
task_queue = queue.Queue()
# 接收结果的队列
result_queue = queue.Queue()
# 从BaseManager继承的QueueManager：
class QueueManager(BaseManager):
pass
# 把两个Queue都注册到网络上，callable参数关联了Queue对象
QueueManager.register('get_task_queue', callable=lambda: task_queue)
QueueManager.register('get_result_queue', callable=lambda: result_queue)
# 绑定端口5000，设置验证码‘abc’
manager = QueueManager(address=('', 5000), authkey=b'abc')
# 启动Queue：
manager.start()
# 获得通过网络访问的Queue对象
task = manager.get_task_queue()
result = manager.get_result_queue()
# 放几个任务进去：
for i in range(10):
n = random.randint(0, 10000)
print('Put task %d...' % n)
task.put(n)
# 从result队列读取结果：
print('Try get result...')
for i in range(10):
r = result.get(timeout=10)
print('Result: %s' % r)
# 关闭
manager.shutdown()
print('master exit.')

报错（一）：
执行报错“_pickle.PicklingError: Can’t pickle <function at 0x0000021EED298438>: attribute lookup on main failed”，具体报错内容如下：

"D:Program Filespython-3.7.9python.exe" D:/py/test-three/test_07.py
Traceback (most recent call last):
File "D:/py/test-three/test_07.py", line 22, in <module>
manager.start()
File "D:Program Filespython-3.7.9libmultiprocessingmanagers.py", line 563, in start
self._process.start()
File "D:Program Filespython-3.7.9libmultiprocessingprocess.py", line 112, in start
self._popen = self._Popen(self)
File "D:Program Filespython-3.7.9libmultiprocessingcontext.py", line 322, in _Popen
return Popen(process_obj)
File "D:Program Filespython-3.7.9libmultiprocessingpopen_spawn_win32.py", line 89, in __init__
reduction.dump(process_obj, to_child)
File "D:Program Filespython-3.7.9libmultiprocessingreduction.py", line 60, in dump
ForkingPickler(file, protocol).dump(obj)
_pickle.PicklingError: Can't pickle <function <lambda> at 0x0000021EED298438>: attribute lookup <lambda> on __main__ failed

首先查看报错内容，寻找错误源头‘manager.start()’，无法启动Queue，根据“_pickle.PicklingError: Can’t pickle <function at 0x0000021EED298438>: attribute lookup on main failed”，lambda表达式无法序列化，参考文章【https://blog.csdn.net/tpc4289/article/details/79280659/】，自定义函数，修改部分代码如下：

# 自定义函数return_tast_queue
def return_task_queue():
global task_queue
return task_queue
# 自定义函数return_result_queue
def return_result_queue():
global result_queue
return result_queue
# 把两个Queue都注册到网络上，callable参数关联了Queue对象
QueueManager.register('get_task_queue', callable=return_task_queue)
QueueManager.register('get_result_queue', callable=return_result_queue)

执行修改后的代码，再次报错，报错内容变化，说明上一个问题解决，新增其他问题。
报错（二）：
is not going to be frozen to produce an executable.‘’')……具体报错内容为：

"D:Program Filespython-3.7.9python.exe" D:/py/test-three/test_07.py
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "D:Program Filespython-3.7.9libmultiprocessingspawn.py", line 105, in spawn_main
exitcode = _main(fd)
File "D:Program Filespython-3.7.9libmultiprocessingspawn.py", line 114, in _main
prepare(preparation_data)
File "D:Program Filespython-3.7.9libmultiprocessingspawn.py", line 225, in prepare
_fixup_main_from_path(data['init_main_from_path'])
File "D:Program Filespython-3.7.9libmultiprocessingspawn.py", line 277, in _fixup_main_from_path
run_name="__mp_main__")
File "D:Program Filespython-3.7.9librunpy.py", line 263, in run_path
pkg_name=pkg_name, script_name=fname)
File "D:Program Filespython-3.7.9librunpy.py", line 96, in _run_module_code
mod_name, mod_spec, pkg_name, script_name)
File "D:Program Filespython-3.7.9librunpy.py", line 85, in _run_code
exec(code, run_globals)
File "D:pytest-threetest_07.py", line 32, in <module>
manager.start()
File "D:Program Filespython-3.7.9libmultiprocessingmanagers.py", line 563, in start
self._process.start()
File "D:Program Filespython-3.7.9libmultiprocessingprocess.py", line 112, in start
self._popen = self._Popen(self)
File "D:Program Filespython-3.7.9libmultiprocessingcontext.py", line 322, in _Popen
return Popen(process_obj)
File "D:Program Filespython-3.7.9libmultiprocessingpopen_spawn_win32.py", line 46, in __init__
prep_data = spawn.get_preparation_data(process_obj._name)
File "D:Program Filespython-3.7.9libmultiprocessingspawn.py", line 143, in get_preparation_data
_check_not_importing_main()
File "D:Program Filespython-3.7.9libmultiprocessingspawn.py", line 136, in _check_not_importing_main
is not going to be frozen to produce an executable.''')
RuntimeError:
An attempt has been made to start a new process before the
current process has finished its bootstrapping phase.
This probably means that you are not using fork to start your
child processes and you have forgotten to use the proper idiom
in the main module:
if __name__ == '__main__':
freeze_support()
...
The "freeze_support()" line can be omitted if the program
is not going to be frozen to produce an executable.

这个报错内容很明显，需要添加“if name == ‘main’:”，具体原理可参考文章【https://blog.csdn.net/shenfuli/article/details/103969964】

报错（三）：

"D:Program Filespython-3.7.9python.exe" D:/py/test-three/test_08.py
Traceback (most recent call last):
File "D:/py/test-three/test_08.py", line 36, in <module>
task = manager.get_task_queue()
File "D:Program Filespython-3.7.9libmultiprocessingmanagers.py", line 724, in temp
token, exp = self._create(typeid, *args, **kwds)
File "D:Program Filespython-3.7.9libmultiprocessingmanagers.py", line 607, in _create
conn = self._Client(self._address, authkey=self._authkey)
File "D:Program Filespython-3.7.9libmultiprocessingconnection.py", line 492, in Client
c = SocketClient(address)
File "D:Program Filespython-3.7.9libmultiprocessingconnection.py", line 620, in SocketClient
s.connect(address)
OSError: [WinError 10049] 在其上下文中，该请求的地址无效。
进程已结束,退出代码1

此处就是照着源代码输入时，丢失ip，需要填入自己的IP，cmd输入‘ipconfig’查看本机ip，填入此段代码：manager = QueueManager(address=(‘此处填入本机IP’, 5000), authkey=b’abc’)

修改后的正确代码如下：

import random, queue
from multiprocessing.managers import BaseManager
# 发送任务的队列：
task_queue = queue.Queue()
# 接收结果的队列
result_queue = queue.Queue()
def return_task_queue():
global task_queue
return task_queue
def return_result_queue():
global result_queue
return result_queue
# 从BaseManager继承的QueueManager：
class QueueManager(BaseManager):
pass
if __name__ == '__main__':
# 把两个Queue都注册到网络上，callable参数关联了Queue对象
QueueManager.register('get_task_queue', callable=return_task_queue)
QueueManager.register('get_result_queue', callable=return_result_queue)
# 绑定端口5000，设置验证码‘abc’
manager = QueueManager(address=('本机ip', 5000), authkey=b'abc')
# 启动Queue：
manager.start()
# 获得通过网络访问的Queue对象
task = manager.get_task_queue()
result = manager.get_result_queue()
# 放几个任务进去：
for i in range(10):
n = random.randint(0, 10000)
print('Put task %d...' % n)
task.put(n)
# 从result队列读取结果：
print('Try get result...')
for i in range(10):
r = result.get(timeout=10)
print('Result: %s' % r)
# 关闭
manager.shutdown()
print('master exit.')