RuntimeError: DataLoader worker (pid 5657)

最新推荐文章于 2024-07-31 15:45:57 发布

dashan8608

最新推荐文章于 2024-07-31 15:45:57 发布

阅读量1.6w

点赞数 4

本文链接：https://blog.csdn.net/dashan8608/article/details/85090476

版权

RuntimeError: DataLoader worker (pid 5657) is killed by signal: Killed. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace.

I encountered into the following problem when I was trying to read a batch （32）of relatively large data sample with multi-threaded DataLoader (with num_workers=8 for example).
I have tried increasing the shared memory of ubuntu but did not work.
It will run without the num_workers argument, but it is too slow to learn from a large data set with single thread.

参考了一些答案后改成num-worker为4 batch-size为20

In my case, I find that it caused by out of memory of machine(note that , not GPU memory). To test that when run your code, you can use command top to see the whether the memory and swap memory are used up. If so , it is the memory problem. You can use a small batch, use a small num_workers, expand your swap memory,etc. Any way could decrease your memory load would be helpful.