训练过程中,内存逐渐升高,最后因内存不足导致训练的进程意外结束:
结束时显示的问题:
numpy.core._exceptions.MemoryError: Unable to allocate xxx GB
运行时的截图:
问题分析与解决:
- tensorflow本身的问题:fit_generator()在存在class_weight时会导致内存泄漏问题:
- 问题解决:
- stackoverflow原话:
- For any future users, there seems to be a bug in the nightly build which seems to be fixed in the subsequent nightly builds. More details here in bug report.
回答:https://github.com/tensorflow/tensorflow/issues/31253
- 通过清理软件定时清理解决:
鲁大师有定时清理功能,差不多一两个小时清理一次