求助：RuntimeError: CUDA error: unknown error

向山

已于 2024-07-08 11:30:01 修改

阅读量408

点赞数 3

文章标签：深度学习人工智能计算机视觉

于 2024-07-08 10:23:09 首次发布

本文链接：https://blog.csdn.net/weixin_45273354/article/details/140261301

版权

以前能跑完训练的代码，上周开始跑到一半就会报错，batch_size调小了一半也不行。

Traceback (most recent call last):
File "D:\re_id\projects\MSCMNet-main\train.py", line 452, in <module>
if epoch > 0 and epoch % 1 == 0:
^^^^^^^^^^^^
File "D:\re_id\projects\MSCMNet-main\train.py", line 303, in train

File "D:\ProgramData\anaconda3\Lib\site-packages\torch\amp\grad_scaler.py", line 453, in step
retval = self._maybe_opt_step(optimizer, optimizer_state, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\ProgramData\anaconda3\Lib\site-packages\torch\amp\grad_scaler.py", line 350, in _maybe_opt_step
if not sum(v.item() for v in optimizer_state["found_inf_per_device"].values()):
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\ProgramData\anaconda3\Lib\site-packages\torch\amp\grad_scaler.py", line 350, in <genexpr>
if not sum(v.item() for v in optimizer_state["found_inf_per_device"].values()):
^^^^^^^^
RuntimeError: CUDA error: unknown error
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.