Pytorch报错 CUDA error: device-side assert triggered

最新推荐文章于 2023-12-04 22:13:21 发布
最新推荐文章于 2023-12-04 22:13:21 发布
阅读量5.1k 收藏 5
点赞数 7
分类专栏： Pytorch深度学习 八阿哥图鉴 文章标签： pytorch python cuda bug gpu
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/veritasalice/article/details/111917185
版权

错误信息：

RuntimeError: CUDA error: device-side assert triggered
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:108: cunn_ClassNLLCriterion_updateOutput_kernel: block: [0,0,0], thread: [1,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:108: cunn_ClassNLLCriterion_updateOutput_kernel: block: [0,0,0], thread: [2,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:108: cunn_ClassNLLCriterion_updateOutput_kernel: block: [0,0,0], thread: [4,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:108: cunn_ClassNLLCriterion_updateOutput_kernel: block: [0,0,0], thread: [5,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:108: cunn_ClassNLLCriterion_updateOutput_kernel: block: [0,0,0], thread: [8,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:108: cunn_ClassNLLCriterion_updateOutput_kernel: block: [0,0,0], thread: [14,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:108: cunn_ClassNLLCriterion_updateOutput_kernel: block: [0,0,0], thread: [15,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:108: cunn_ClassNLLCriterion_updateOutput_kernel: block: [0,0,0], thread: [16,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:108: cunn_ClassNLLCriterion_updateOutput_kernel: block: [0,0,0], thread: [17,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:108: cunn_ClassNLLCriterion_updateOutput_kernel: block: [0,0,0], thread: [18,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:108: cunn_ClassNLLCriterion_updateOutput_kernel: block: [0,0,0], thread: [0,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:108: cunn_ClassNLLCriterion_updateOutput_kernel: block: [0,0,0], thread: [1,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:108: cunn_ClassNLLCriterion_updateOutput_kernel: block: [0,0,0], thread: [2,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:108: cunn_ClassNLLCriterion_updateOutput_kernel: block: [0,0,0], thread: [4,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:108: cunn_ClassNLLCriterion_updateOutput_kernel: block: [0,0,0], thread: [5,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:108: cunn_ClassNLLCriterion_updateOutput_kernel: block: [0,0,0], thread: [8,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:108: cunn_ClassNLLCriterion_updateOutput_kernel: block: [0,0,0], thread: [11,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:108: cunn_ClassNLLCriterion_updateOutput_kernel: block: [0,0,0], thread: [14,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:108: cunn_ClassNLLCriterion_updateOutput_kernel: block: [0,0,0], thread: [15,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:108: cunn_ClassNLLCriterion_updateOutput_kernel: block: [0,0,0], thread: [16,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:108: cunn_ClassNLLCriterion_updateOutput_kernel: block: [0,0,0], thread: [19,0,0] Assertion `t >= 0 && t < n_classes` failed.
terminate called after throwing an instance of 'c10::Error'
  what():  CUDA error: device-side assert triggered
Exception raised from create_event_internal at /pytorch/c10/cuda/CUDACachingAllocator.cpp:687 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x42 (0x7fcb23d5c8b2 in /home/*/anaconda3/lib/python3.7/site-packages/torch/lib/libc10.so)
frame #1: c10::cuda::CUDACachingAllocator::raw_delete(void*) + 0xad2 (0x7fcb23fae952 in /home/*/anaconda3/lib/python3.7/site-packages/torch/lib/libc10_cuda.so)
frame #2: c10::TensorImpl::release_resources() + 0x4d (0x7fcb23d47b7d in /home/*/anaconda3/lib/python3.7/site-packages/torch/lib/libc10.so)
frame #3: <unknown function> + 0x5ff5ba (0x7fcb727405ba in /home/*/anaconda3/lib/python3.7/site-packages/torch/lib/libtorch_python.so)
frame #4: <unknown function> + 0x5ff666 (0x7fcb72740666 in /home/*/anaconda3/lib/python3.7/site-packages/torch/lib/libtorch_python.so)
<omitting python frames>
frame #35: __libc_start_main + 0xe7 (0x7fcb75ab7b97 in /lib/x86_64-linux-gnu/libc.so.6)

terminate called after throwing an instance of 'c10::Error'
  what():  CUDA error: device-side assert triggered
Exception raised from create_event_internal at /pytorch/c10/cuda/CUDACachingAllocator.cpp:687 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x42 (0x7fceacad88b2 in /home/*/anaconda3/lib/python3.7/site-packages/torch/lib/libc10.so)
frame #1: c10::cuda::CUDACachingAllocator::raw_delete(void*) + 0xad2 (0x7fceacd2a952 in /home/*/anaconda3/lib/python3.7/site-packages/torch/lib/libc10_cuda.so)
frame #2: c10::TensorImpl::release_resources() + 0x4d (0x7fceacac3b7d in /home/*/anaconda3/lib/python3.7/site-packages/torch/lib/libc10.so)
frame #3: <unknown function> + 0xaac8b9 (0x7fceeac528b9 in /home/*/anaconda3/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
frame #4: <unknown function> + 0x27cfba6 (0x7fceec975ba6 in /home/*/anaconda3/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
frame #5: torch::autograd::deleteNode(torch::autograd::Node*) + 0x112 (0x7fceecda57b2 in /home/*/anaconda3/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
frame #6: <unknown function> + 0x485319 (0x7fcefb342319 in /home/*/anaconda3/lib/python3.7/site-packages/torch/lib/libtorch_python.so)
frame #7: c10::TensorImpl::release_resources() + 0x20 (0x7fceacac3b50 in /home/*/anaconda3/lib/python3.7/site-packages/torch/lib/libc10.so)
frame #8: <unknown function> + 0x5ff5ba (0x7fcefb4bc5ba in /home/*/anaconda3/lib/python3.7/site-packages/torch/lib/libtorch_python.so)
frame #9: <unknown function> + 0x5ff666 (0x7fcefb4bc666 in /home/*/anaconda3/lib/python3.7/site-packages/torch/lib/libtorch_python.so)
frame #10: <unknown function> + 0x1a184f (0x55f609aa184f in /home/*/anaconda3/bin/python)
frame #11: <unknown function> + 0x10db9b (0x55f609a0db9b in /home/*/anaconda3/bin/python)
frame #12: <unknown function> + 0x1a184f (0x55f609aa184f in /home/*/anaconda3/bin/python)
frame #13: <unknown function> + 0x123b73 (0x55f609a23b73 in /home/*/anaconda3/bin/python)
frame #14: _PyGC_CollectNoFail + 0x2a (0x55f609b1bfda in /home/*/anaconda3/bin/python)
frame #15: PyImport_Cleanup + 0x29c (0x55f609aa7bcc in /home/*/anaconda3/bin/python)
frame #16: Py_FinalizeEx + 0x67 (0x55f609b23087 in /home/*/anaconda3/bin/python)
frame #17: <unknown function> + 0x235f93 (0x55f609b35f93 in /home/*/anaconda3/bin/python)
frame #18: _Py_UnixMain + 0x3c (0x55f609b362bc in /home/*/anaconda3/bin/python)
frame #19: __libc_start_main + 0xe7 (0x7fcefe833b97 in /lib/x86_64-linux-gnu/libc.so.6)
frame #20: <unknown function> + 0x1db062 (0x55f609adb062 in /home/*/anaconda3/bin/python)

Aborted (core dumped)

错误原因：

pytorch要求label必须是0到n的连续整数，我出错的原因是二分类label用的是0和2这种不连续的整数，改成0,1即可解决。

Dr. 卷心菜
关注
  • 7
    点赞
  • 踩
  • 5
    收藏
    觉得还不错? 一键收藏
  • 打赏
    打赏
  • 0
    评论
专栏目录
解决pytorch报错:AssertionError: Invalid device id的问题
01-21
在服务器上训练的网络放到本地台式机进行infer，结果出现报错： AssertionError: Invalid device id ...以上这篇解决pytorch报错:AssertionError: Invalid device id的问题就是小编分享给大家的全部内容了，
PyTorch 错误 RuntimeError: CUDA error: device-side assert triggered
最新发布
不会写代码的码农
05-21 282
检查一下数据集标签和训练时的标签是否一致。
pytorch CUDA error: device-side assert triggered
Andrew_SJ的博客
11-27 222
在做多分类问题的实验时出现了错误，提示： pytorch CUDA error: device-side assert triggered 搜索发现均说明是标签个数不匹配的问题，仔细查了一下，原来是在对数据集做新的分类后，数据类别为7，而模型没有更改，最后一层还是原来的6个神经元，更改之后就可以了。 ...
Debug Pytorch: RuntimeError: CUDA error: device-side assert triggered
yuetan的博客
06-07 360
我这里只要to device就报错了。把to device注释掉，可以解决问题。首先采用普通的torch.randint测试，发现报同样的错误，可知是模型的问题。或者能重启也就解决了。
pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:108: block: [0,0,0], thread: [11,0,0] Assertion `t ＞= 0
知识搬运者
12-19 2603
调用交叉熵损失函数nn.CrossEntropyLoss()的时候报错。 原因: nn.CrossEntropyLoss(input, target)中, input的列数和target不匹配 举例来说, 如果是个5分类问题, 那么input的形状是batch_size * 5, target中每个元素的取值范围是{0,1,2,3,4}, 也就是[0, 类别数量减1]。 所以对于n分类问题来说, 如果target中某个元素的取值不在[0,n-1]中就会...
PyTorch 错误 RuntimeError: CUDA error: device-side assert triggered
flyfish
03-05 6万+
PyTorch 错误 RuntimeError: CUDA error: device-side assert triggered flyfish 具体提示如下 /pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:108: cunn_ClassNLLCriterion_updateOutput_kernel: block: [0,0,0], thread: [2,0,0] Assertion `t >= 0 && t < n_classe
Bert分类模型训练报错：RuntimeError: CUDA error: device-side assert triggered
凝眸伏笔的博客
04-25 1622
问题： 训练bert分类模型时候，报错如下： <torch.utils.data.dataloader.DataLoader object at 0x7f63fe59a588> Epoch [1/50] /pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:108: cunn_ClassNLLCriterion_updateOutput_kernel: block: [0,0,0], thread: [0,0,0] Assertion `t >=
Pytorch报错：RuntimeError: cuda runtime error (710) : device-side assert triggered
weixin_42138934的博客
07-31 8685
最近在研究分类网络MSD-Net，打算跑cifar10和cifar100数据集来复现一下论文的研究结果。 GitHub：MSD-Net的Pytorch版 总体代码写完后，跑cifar10数据集非常完美，分类准确率比ResNet和DenseNet都有提升。但是跑cifar100数据集的时候就报了题目的错 C:/w/1/s/windows/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: block: [0,0,0], thread: [31,0,0] Asse
报错CUDA error: device-side assert triggered
weixin_42535423的博客
07-18 2193
CUDA error: device-side assert triggered /tmp/pip-req-build-ocx5vxk7/aten/src/THCUNN/ClassNLLCriterion.cu:106: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctyp
yolov5-3.1版本训练时可能会报错及修改方法汇总
08-20
二、 TypeError: can't convert cuda:0 device type tensor to numpy. Use Tensor.cpu() to copy the tensor to host memory first. 在 yolov5-3.1 版本训练时，可能会出现上述报错。这是因为 PyTorch 中的 Tensor ...
RandLA-Net-pytorch:RandLA-Net（https的Pytorch实施
05-23
我们的Pytorch实施 0.529 每课时 密欧 车 自行车 摩托车 卡车 其他车辆 人 骑自行车的人 电单车司机 路 停车处 人行道 其他地面 建造 栅栏 植被 树干 地形 极 交通标志 52.9 0.919 0.122 0.290 0.660 0.444 ...
pytorch报错：Process finished with exit code -1073741819 (0xC0000005)
12-21
网上各种解决方法，但是我都试了不可以，我实验发现如果不采用gpu环境的pytorch程序无报错，采用使用gpu的pytoch程序报错，采用gpu的tensroflow和keras不报错。这就让我很疑惑，为什么只有采用gpu的pytorch程序才会...
云主机vit bug处理（自用）
cubic_h的博客
12-04 394
方法： rmdir xxxxx/.ipynb_checkpoints，也即将自己数据集路径下的.ipynb_checkpoints删去（此文件夹不可见）
pytorch报错：RuntimeError: CUDA error: device-side assert triggered
热门推荐
Lily's Blog
11-29 8万+
训练网络报错：RuntimeError: cuda runtime error (710) : device-side assert triggered at /pytorch/aten/src/THC/generic/THCTensorScatterGather.cu:380 terminate called after throwing an instance of 'c10::Error' ...
PyTorch报错：RuntimeError: CUDA error: device-side assert triggered at /pytorch/aten/src/THC/generic
希望你会喜欢
04-01 1301
训练模型报错 RuntimeError: cuda runtime error (710) : device-side assert triggered at /pytorch/aten/src/THC/generic/THCTensorMath.cu:29 问题解决 原因：标签（label）越界，一般从0开始 方法：更改你的标签值，从0开始 target_names = ['__label__1', '__label__2', '__label__3', '__label__4', '__label__
RuntimeError: CUDA error: device-side assert triggered
weixin_44003026的博客
12-09 1794
RuntimeError: CUDA error: device-side assert triggered CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect. For debugging consider passing CUDA LAUNCH BLOCKING=1 pytorch/aten/src/ATen/native
pytorch奇葩错误
Dongjiuqing的博客
08-26 764
https://blog.csdn.net/zcgyq/article/details/83085028 访问本站观看效果更佳 在pytorch中遇到如下错误： libraries/torch/extra/cunn/lib/THCUNN/ClassNLLCriterion.cu:52: void cunn_ClassNLLCriterion_updateOutput_kernel(Dty...
yolact RuntimeError: CUDA error: device-side assert triggered
08-30
yolact RuntimeError: CUDA error: device-side assert triggered是指在运行yolact模型时出现了CUDA错误，具体为设备端断言触发。根据引用[3]所述，这个错误通常是由于代码中出现了索引越界导致的。然而，定位到报错位置并不能直接找到错误的根本原因，也无法从报错信息中得知具体的解决方案。 为了解决这个问题，可以尝试使用CPU而非GPU运行代码。这样一来，就可以通过报错信息较容易地找到错误的位置和原因。在将代码切换回GPU运行之前，先解决错误，并确保代码不会出现索引越界的问题。 另外，根据引用的建议，还可以通过调整batch_size的大小来减少显存的使用量。此外，使用item()属性来获取torch变量的标量值也可以减少显存的占用。 综上所述，解决yolact RuntimeError: CUDA error: device-side assert triggered的方法是： 1. 尝试使用CPU而非GPU运行代码，通过报错信息定位错误位置和原因，并解决索引越界问题。 2. 调整batch_size的大小以减少显存的使用量。 3. 使用item()属性来获取torch变量的标量值，减少显存的占用。 请根据具体情况尝试这些解决方法，以解决该错误。<span class="em">1</span><span class="em">2</span><span class="em">3</span> #### 引用[.reference_title] - *1* [RuntimeError: CUDA error: device-side assert triggered](https://blog.csdn.net/HPU_CPD/article/details/126592019)[target="_blank" data-report-click={"spm":"1018.2226.3001.9630","extra":{"utm_source":"vip_chatgpt_common_search_pc_result","utm_medium":"distribute.pc_search_result.none-task-cask-2~all~insert_cask~default-1-null.142^v93^chatsearchT3_1"}}] [.reference_item style="max-width: 33.333333333333336%"] - *2* [pytorch模型提示超出内存RuntimeError: CUDA out of memory.](https://download.csdn.net/download/weixin_38552305/14884908)[target="_blank" data-report-click={"spm":"1018.2226.3001.9630","extra":{"utm_source":"vip_chatgpt_common_search_pc_result","utm_medium":"distribute.pc_search_result.none-task-cask-2~all~insert_cask~default-1-null.142^v93^chatsearchT3_1"}}] [.reference_item style="max-width: 33.333333333333336%"] - *3* [RuntimeError: CUDA error: device-side assert triggered Pytorch框架代码运行错误解决方案(亲测有效！...](https://blog.csdn.net/weixin_42112050/article/details/120455407)[target="_blank" data-report-click={"spm":"1018.2226.3001.9630","extra":{"utm_source":"vip_chatgpt_common_search_pc_result","utm_medium":"distribute.pc_search_result.none-task-cask-2~all~insert_cask~default-1-null.142^v93^chatsearchT3_1"}}] [.reference_item style="max-width: 33.333333333333336%"] [ .reference_list ]

“相关推荐”对你有帮助么？

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包

打赏作者

Dr. 卷心菜

你的鼓励将是我创作的最大动力

¥1 ¥2 ¥4 ¥6 ¥10 ¥20
扫码支付：¥1
获取中
扫码支付

您的余额不足，请更换扫码支付或充值

打赏作者

实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值