python process join_为什么多处理.Process.join()挂断?

我以这种方式使用多重处理:import multiprocessing as mp

def worker(thread_id, tasks, results):

tmp_dir = 'temp_for_{}'.format(thread_id)

os.makedirs(tmp_dir)

try:

while not tasks.empty():

data = tasks.get()

response = process_pdf(data, tmp_dir)

results.put(response)

except (keyboardInterrupt, SystemExit):

log.info('Interrupt signal received in thread %s.', thread_id)

except Queue.Empty:

pass

except Exception:

log.error("Unexpected error in %s", thread_id, exc_info=True)

finally:

shutil.rmtree(tmp_dir)

log.info("Thread %s exit", thread_id)

if __name__ == "__main__":

tasks, results = mp.Queue(), mp.Queue()

for record in cursor.select(query):

tasks.put(record)

manager = mp.Manager()

workers = [mp.Process(target=worker, args=(i, tasks, results)) for i in xrange(8)]

for worker in workers:

worker.start()

try:

for worker in workers:

worker.join()

except (KeyboardInterrupt, SystemExit):

log.info('Interrupt signal received in main. Cleaning up main')

finally:

log.info('Got %s results. Saving', results.qsize())

while not results.empty():

cursor.update_one('documents', 'id', results.get())

cursor.close()

以下是我运行此代码时的输出:

^{pr2}$

然后我在等待了一段时间后输入^C,并得到以下输出:^C14:37:16 15/10 INFO: Interrupt signal received in main. Cleaning up main

14:37:16 15/10 INFO: Got 16 results. Saving

我得到了所有线程的回溯:Process Process-9:

Traceback (most recent call last):

File "/usr/lib64/python2.7/multiprocessing/process.py", line 261, in _bootstrap

util._exit_function()

File "/usr/lib64/python2.7/multiprocessing/util.py", line 328, in _exit_function

util._exit_function()

File "/usr/lib64/python2.7/multiprocessing/util.py", line 274, in _run_finalizers

finalizer()

File "/usr/lib64/python2.7/multiprocessing/util.py", line 207, in __call__

res = self._callback(*self._args, **self._kwargs)

File "/usr/lib64/python2.7/multiprocessing/queues.py", line 218, in _finalize_join

thread.join()

File "/usr/lib64/python2.7/threading.py", line 952, in join

thread.join()

File "/usr/lib64/python2.7/threading.py", line 340, in wait

waiter.acquire()

KeyboardInterrupt

为什么要绞死?如果重要的话,我可以添加process_pdf()使用subprocess.Popen()运行一些子流程。在

  • 0
    点赞
  • 0
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值