Pytorch 尝试通过强化cpu使用加快训练和推理速度（二）

简介：这系列第一篇中讲了多线程和多进程的适用范围，写这篇的原因是遇到了进程池嵌套进程池的问题。（情况：我写的代码中使用了进程池，其中一个进程调用它他人写的使用进程池的函数。表现：他的函数直接跳过没有任何报错，最终定位到pool=Pool（x）。）

1. 先举个进程池嵌套进程池的代码例子.（参考后续文献）

import multiprocessing
import multiprocessing.pool
import time

from random import randint

def sleepawhile(t):
    print("Sleeping %i seconds..." % t)
    time.sleep(t)
    return t

def work(num_procs):
    print("Creating %i (daemon) workers and jobs in child." % num_procs)
    pool = multiprocessing.Pool(num_procs)

    result = pool.map(sleepawhile,
        [randint(1, 5) for x in range(num_procs)])
    
    pool.close()
    pool.join()
    return result

def test():
    print("Creating 5 (non-daemon) workers and jobs in main process.")
    pool = multiprocessing.Pool(5)

    result = pool.map(work, [randint(1, 5) for x in range(5)])

    pool.close()
    pool.join()
    print(result)

if __name__ == '__main__':
    test()

运行后报错如下：

AssertionError: daemonic processes are not allowed to have children

修改后如下代码就不报错了（python3.8，不通版本改法可能不同详见参考链接）：

import multiprocessing
import multiprocessing.pool
import time

from random import randint
class NonDaemonPool(multiprocessing.pool.Pool):
    def Process(self, *args, **kwds):
        proc = super(NonDaemonPool, self).Process(*args, **kwds)

        class NonDaemonProcess(proc.__class__):
            """Monkey-patch process to ensure it is never daemonized"""
            @property
            def daemon(self):
                return False

            @daemon.setter
            def daemon(self, val):
                pass

        proc.__class__ = NonDaemonProcess
        return proc


def sleepawhile(t):
    print("Sleeping %i seconds..." % t)
    time.sleep(t)
    return t


def work(num_procs):
    print("Creating %i (daemon) workers and jobs in child." % num_procs)
    pool = multiprocessing.Pool(num_procs)

    result = pool.map(sleepawhile,
                      [randint(1, 5) for x in range(num_procs)])

    pool.close()
    pool.join()
    return result


def test():
    print("Creating 5 (non-daemon) workers and jobs in main process.")
    pool = NonDaemonPool(5)

    result = pool.map(work, [randint(1, 5) for x in range(5)])

    pool.close()
    pool.join()
    print(result)


if __name__ == '__main__':
    test()

 

参考：https://stackoverflow.com/questions/17223301/python-multiprocessing-is-it-possible-to-have-a-pool-inside-of-a-pool，https://stackoverflow.com/questions/6974695/python-process-pool-non-daemonic/8963618#8963618，https://www.cnpython.com/qa/262520