RuntimeError: cuda runtime error (59) : device-side assert triggered at /opt/conda/conda-bld/...

最新推荐文章于 2024-06-21 15:03:56 发布

fanfan121314

最新推荐文章于 2024-06-21 15:03:56 发布

阅读量2.3k

点赞数

分类专栏：技术问题文章标签： python

本文链接：https://blog.csdn.net/fanfan121314/article/details/108725975

版权

技术问题专栏收录该内容

4 篇文章 0 订阅

订阅专栏

看了多篇文章，基本上这个问题就是计算损失函数的时候，label出了问题。
有时候报的错比较简单，就是这一句话：RuntimeError: cuda runtime error (59) : device-side assert triggered at /opt/conda/conda-bld/…，为了更明确可以打印细节，就是在导入模块之前输入

import os
os.environ['CUDA_LAUNCH_BLOCKING'] = "1"

就会显示

cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [26,0,0] Assertion `t >= 0 && t < n_classes` failed.
THCudaCheck FAIL file=/opt/conda/conda-bld/pytorch_1556653114079/work/aten/src/THCUNN/generic/ClassNLLCriterion.cu line=110 error=59 : device-side assert triggered
Traceback (most recent call last):
  File "train_semseg_mine.py", line 320, in <module>
    main(args)
  File "train_semseg_mine.py", line 222, in main
    loss = criterion(seg_pred, target, trans_feat)#璁＄畻鎹熷け
  File "/root/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "/root/project/pointnet++/models/pointnet2_sem_seg.py", line 49, in forward
    total_loss = F.nll_loss(pred, target)#, weight=weight
  File "/root/anaconda3/lib/python3.7/site-packages/torch/nn/functional.py", line 1871, in nll_loss
    ret = torch._C._nn.nll_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index)
RuntimeError: cuda runtime error (59) : device-side assert triggered at /opt/conda/conda-bld/pytorch_1556653114079/work/aten/src/THCUNN/generic/ClassNLLCriterion.cu:110

接下来为了确定label哪里出了问题，就可以打印一下label。有时候我们的数据比较多，需要进行不重复计数查看。

#不重复计数查看数据的label，也就是下面的target
    for i, (input, target) in enumerate(TRAIN_DATASET):
        target = target.numpy()#因为target一开始是张量形式，这里把它转换成array
        target = np.unique(target)
        print("target: {}".format(target))

我的打印结果有: [-1, 0, 1, 2, 3, 4, 5]
这样就不对了，每个分量应该是在0-6之间，这里6表示（种类的个数-1）。所以就把你的标签放在[0, 种类个数-1]之间就好了。比如说我上面这个错误，就给label+1就行了

参考文章
RuntimeError: cuda runtime error (59) : device-side assert triggered