【Python fits】Error: Too many open files

循环打开了好多个Fits文件读数据，然后突然提示“你打开太多文件啦”，后来就加了一行hdu.close(), 但是并没有什么乱用

查了一下说明才发现，是因为用open()打开后是存储在mmap中的，为了缓解内存，关闭hdu也没有用，文件还是开着呢

而怎么关闭文件呢，需要等到numpy觉得你并不需要这个data之后“自动”关闭，啊，如果它一直觉得你要用。。。。嗯。。。

解决方法很简单，告诉它，“我不用这个data啦”

所以，不是加一句hdu.close()，而是del hdu_data     del hdu.data

测试了一下，这样解决的是fits文件比较大，里面包含了太多的hdu，但是对于fits文件比较多的情况不适用

fits文件太多的时候，用hdu = fits.open(fits_name,memmap=False)直接不存在mmap中

 

原文和链接在下面

http://docs.astropy.org/en/stable/io/fits/appendix/faq.html

I’m opening many FITS files in a loop and getting OSError: Too many open files

Say you have some code like:

The details may differ, but the qualitative point is that the data to many HDUs and/or FITS files are being accessed in a loop. This may result in an exception like:

 

As explained in the note on working with large files, because Astropy uses mmap by default to read the data in a FITS file, even if you correctly close a file with HDUList.close a handle is kept open to that file so that the memory-mapped data array can still be continued to be read transparently.

The way Numpy supports mmap is such that the file mapping is not closed until the overlying ndarray object has no references to it and is freed memory. However, when looping over a large number of files (or even just HDUs) rapidly, this may not happen immediately. Or in some cases if the HDU object persists, the data array attached to it may persist too. The easiest workaround is to manually delete the .data attribute on the HDU object so that the ndarray reference is freed and the mmap can be closed:

In some extreme cases files are opened and closed fast enough that Python’s garbage collector does not free them (and hence free the file handles) often enough. To mitigate this your code can manually force a garbage collection by calling gc.collect() at the end of the loop.

In a future release it will be easier to automatically perform this sort of cleanup when closing FITS files, where needed.

 

转载于:https://www.cnblogs.com/clementinadoo/p/9675653.html