python self太多_python multiprocessing.Pool太多文件打开日志文件

这是关于Too many files open with multiprocessing.Pool

我有类似的问题.我的设置是运行此简单脚本的四核Ubuntu(运行python 2.7 with pathos == 0.2a1.dev,pathos仅用于允许mp map与类方法一起使用)为每个进程记录一个单独的文件.

python为什么不关闭记录器文件？？？

这是代码…

import multiprocessing

from pathos.multiprocessing import ProcessingPool

import logging

class CreateLogger(object):

def create_logger(self, i):

i = str(i)

logger = logging.getLogger(i)

hdlr = logging.FileHandler('/tmp/test/{0}.log'.format(i))

formatter = logging.Formatter('%(asctime)s %(levelname)s %(message)s')

hdlr.setFormatter(formatter)

logger.addHandler(hdlr)

logger.setLevel(logging.WARNING)

logger.info('{0}-test'.format(i))

logger.propagate = 0

if __name__ == '__main__':

cl = CreateLogger()

ilist = range(30000)

pool = ProcessingPool(multiprocessing.cpu_count())

pool.map(cl.create_logger, ilist)

错误是：

pydev debugger: starting (pid: 21825)

Traceback (most recent call last):

File "/opt/eclipse/plugins/org.python.pydev_3.8.0.201409251235/pysrc/pydevd.py", line 2183, in

globals = debugger.run(setup['file'], None, None)

File "/opt/eclipse/plugins/org.python.pydev_3.8.0.201409251235/pysrc/pydevd.py", line 1622, in run

pydev_imports.execfile(file, globals, locals) # execute the script

File "/home/amit/workspace/amit/device_polling/tests/simple_mp_test.py", line 21, in

pool.map(cl.create_logger, ilist)

File "/miniconda/envs/test/lib/python2.7/site-packages/pathos-0.2a1.dev-py2.7.egg/pathos/multiprocessing.py", line 123, in map

return _pool.map(star(f), zip(*args)) # chunksize

File "/miniconda/envs/test/lib/python2.7/site-packages/processing/pool.py", line 130, in map

return self.mapAsync(func, iterable, chunksize).get()

File "/miniconda/envs/test/lib/python2.7/site-packages/processing/pool.py", line 373, in get

raise self._value

IOError: [Errno 24] Too many open files: '/tmp/test/15336.log'

您会看到太多文件打开错误始于15336.在调试该脚本时,我看到有16K文件正在打开此脚本

amit@sharknado:/tmp/test$lsof -u amit | grep test | wc -l

lsof: WARNING: can't stat() fuse.gvfsd-fuse file system /run/user/112/gvfs

Output information may be incomplete.

16622

amit@sharknado:/tmp/test$