以前能跑完训练的代码,上周开始跑到一半就会报错,batch_size调小了一半也不行。
Traceback (most recent call last):
File "D:\re_id\projects\MSCMNet-main\train.py", line 452, in <module>
if epoch > 0 and epoch % 1 == 0:
^^^^^^^^^^^^
File "D:\re_id\projects\MSCMNet-main\train.py", line 303, in train
File "D:\ProgramData\anaconda3\Lib\site-packages\torch\amp\grad_scaler.py", line 453, in step
retval = self._maybe_opt_step(optimizer, optimizer_state, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\ProgramData\anaconda3\Lib\site-packages\torch\amp\grad_scaler.py", line 350, in _maybe_opt_step
if not sum(v.item() for v in optimizer_state["found_inf_per_device"].values()):
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\ProgramData\anaconda3\Lib\site-packages\torch\amp\grad_scaler.py", line 350, in <genexpr>
if not sum(v.item() for v in optimizer_state["found_inf_per_device"].values()):
^^^^^^^^
RuntimeError: CUDA error: unknown error
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.