报错如下:
Traceback (most recent call last):
[338] File "/yj/code/global_cover_and_landmark/train.py", line 193, in <module>
[339] main(args)
[340] File "/yj/code/global_cover_and_landmark/train.py", line 164, in main
[341] train(args=args, epoch=epoch, trainloader=trainloader, net=net, optimizer=optimizer,
[342] File "/yj/code/global_cover_and_landmark/train.py", line 59, in train
[343] total_loss.backward()
[344] File "/usr/local/lib/python3.8/dist-packages/torch/_tensor.py", line 363, in backward
[345] torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
[346] File "/usr/local/lib/python3.8/dist-packages/torch/autograd/__init__.py", line 173, in backward
[347] Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass
[348] RuntimeError: CUDA error: device-side assert triggered
[349] CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
[350] For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
解决方法:
在train.py的开头加上两行代码,其实是指定只使用一张卡
import os
os.environ['CUDA_VISIBLE_DEVICES'] = '0'
os.environ['CUDA_LAUNCH_BLOCKING'] = '1'
查找解决的时候,发现还有很多其他原因导致的,欢迎大家补充,嘻嘻嘻