RuntimeError: CUDA error: device-side assert triggered CUDA kernel errors might be asynchronously

最新推荐文章于 2024-03-18 08:57:33 发布
最新推荐文章于 2024-03-18 08:57:33 发布
阅读量4.9k 收藏 3
点赞数 3
分类专栏： 学习笔记 文章标签： 深度学习 python 人工智能
版权声明：本文为博主原创文章，遵循 CC 4.0 BY 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/qq_35291087/article/details/125568988
版权

运行分类任务的时候出现如下报错：

/opt/conda/conda-bld/pytorch_1646755861072/work/aten/src/ATen/native/cuda/Loss.cu:115: operator(): block: [0,0,0], thread: [0,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/opt/conda/conda-bld/pytorch_1646755861072/work/aten/src/ATen/native/cuda/Loss.cu:115: operator(): block: [0,0,0], thread: [1,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/opt/conda/conda-bld/pytorch_1646755861072/work/aten/src/ATen/native/cuda/Loss.cu:115: operator(): block: [0,0,0], thread: [2,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/opt/conda/conda-bld/pytorch_1646755861072/work/aten/src/ATen/native/cuda/Loss.cu:115: operator(): block: [0,0,0], thread: [3,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/opt/conda/conda-bld/pytorch_1646755861072/work/aten/src/ATen/native/cuda/Loss.cu:115: operator(): block: [0,0,0], thread: [4,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/opt/conda/conda-bld/pytorch_1646755861072/work/aten/src/ATen/native/cuda/Loss.cu:115: operator(): block: [0,0,0], thread: [5,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/opt/conda/conda-bld/pytorch_1646755861072/work/aten/src/ATen/native/cuda/Loss.cu:115: operator(): block: [0,0,0], thread: [6,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/opt/conda/conda-bld/pytorch_1646755861072/work/aten/src/ATen/native/cuda/Loss.cu:115: operator(): block: [0,0,0], thread: [7,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/opt/conda/conda-bld/pytorch_1646755861072/work/aten/src/ATen/native/cuda/Loss.cu:115: operator(): block: [0,0,0], thread: [8,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/opt/conda/conda-bld/pytorch_1646755861072/work/aten/src/ATen/native/cuda/Loss.cu:115: operator(): block: [0,0,0], thread: [9,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/opt/conda/conda-bld/pytorch_1646755861072/work/aten/src/ATen/native/cuda/Loss.cu:115: operator(): block: [0,0,0], thread: [10,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/opt/conda/conda-bld/pytorch_1646755861072/work/aten/src/ATen/native/cuda/Loss.cu:115: operator(): block: [0,0,0], thread: [11,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/opt/conda/conda-bld/pytorch_1646755861072/work/aten/src/ATen/native/cuda/Loss.cu:115: operator(): block: [0,0,0], thread: [12,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/opt/conda/conda-bld/pytorch_1646755861072/work/aten/src/ATen/native/cuda/Loss.cu:115: operator(): block: [0,0,0], thread: [13,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/opt/conda/conda-bld/pytorch_1646755861072/work/aten/src/ATen/native/cuda/Loss.cu:115: operator(): block: [0,0,0], thread: [14,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/opt/conda/conda-bld/pytorch_1646755861072/work/aten/src/ATen/native/cuda/Loss.cu:115: operator(): block: [0,0,0], thread: [15,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/opt/conda/conda-bld/pytorch_1646755861072/work/aten/src/ATen/native/cuda/Loss.cu:115: operator(): block: [0,0,0], thread: [16,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/opt/conda/conda-bld/pytorch_1646755861072/work/aten/src/ATen/native/cuda/Loss.cu:115: operator(): block: [0,0,0], thread: [17,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/opt/conda/conda-bld/pytorch_1646755861072/work/aten/src/ATen/native/cuda/Loss.cu:115: operator(): block: [0,0,0], thread: [18,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/opt/conda/conda-bld/pytorch_1646755861072/work/aten/src/ATen/native/cuda/Loss.cu:115: operator(): block: [0,0,0], thread: [19,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/opt/conda/conda-bld/pytorch_1646755861072/work/aten/src/ATen/native/cuda/Loss.cu:115: operator(): block: [0,0,0], thread: [20,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/opt/conda/conda-bld/pytorch_1646755861072/work/aten/src/ATen/native/cuda/Loss.cu:115: operator(): block: [0,0,0], thread: [21,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/opt/conda/conda-bld/pytorch_1646755861072/work/aten/src/ATen/native/cuda/Loss.cu:115: operator(): block: [0,0,0], thread: [22,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/opt/conda/conda-bld/pytorch_1646755861072/work/aten/src/ATen/native/cuda/Loss.cu:115: operator(): block: [0,0,0], thread: [23,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/opt/conda/conda-bld/pytorch_1646755861072/work/aten/src/ATen/native/cuda/Loss.cu:115: operator(): block: [0,0,0], thread: [24,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/opt/conda/conda-bld/pytorch_1646755861072/work/aten/src/ATen/native/cuda/Loss.cu:115: operator(): block: [0,0,0], thread: [25,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/opt/conda/conda-bld/pytorch_1646755861072/work/aten/src/ATen/native/cuda/Loss.cu:115: operator(): block: [0,0,0], thread: [26,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/opt/conda/conda-bld/pytorch_1646755861072/work/aten/src/ATen/native/cuda/Loss.cu:115: operator(): block: [0,0,0], thread: [27,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/opt/conda/conda-bld/pytorch_1646755861072/work/aten/src/ATen/native/cuda/Loss.cu:115: operator(): block: [0,0,0], thread: [28,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/opt/conda/conda-bld/pytorch_1646755861072/work/aten/src/ATen/native/cuda/Loss.cu:115: operator(): block: [0,0,0], thread: [29,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/opt/conda/conda-bld/pytorch_1646755861072/work/aten/src/ATen/native/cuda/Loss.cu:115: operator(): block: [0,0,0], thread: [30,0,0] Assertion `input_val >= zero && input_val <= one` failed.
/opt/conda/conda-bld/pytorch_1646755861072/work/aten/src/ATen/native/cuda/Loss.cu:115: operator(): block: [0,0,0], thread: [31,0,0] Assertion `input_val >= zero && input_val <= one` failed.


RuntimeError: CUDA error: device-side assert triggered
CUDA kernel errors might be asynchronously reported at some other API call,
so the stacktrace below might be incorrect.

真，很烦，因为这是第二次出现这个报错了，更烦的是上次出现这个报错的时候我是怎么修复的我已经不记得了！现在又需要重新整，oh No！故等下找到修复的法门一定要好好记录一番！

  • 记起来了！上次报这个错的原因是做tensor.log()运算的时候，tensor中有很多负数，因此我把tensor用softmax处理一下然后就修复了！！
  • 但是这次的原因还没找到！现在看来应该还是数据的问题，我看github中说是对数组执行索引操作的时候，索引超出了数组本身的数据量，因此出现错误，所以要去定位是什么数据索引超出界限了！
  • 最后发现是梯度爆炸的原因，我把出错那一行代码的前面的tensor值打印出来，发现到后面出现了Nan值，排查后发现是我直接把乘了注意力权重的特征值拿来做后续任务，这就导致后面预测值出现Nan，于是我做了个残差结构：w*F+F(w是权重，F是特征),然后就好了。
李白和野猪
关注
  • 3
    点赞
  • 踩
  • 3
    收藏
    觉得还不错? 一键收藏
  • 1
    评论
专栏目录
Bug小能手系列(python)_13: RuntimeError: CUDA error: device-side assert triggered CUDA kernel errors might
走在深度学习前沿的小宋
10-30 6348
RuntimeError: CUDA error: device-side assert triggered CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1. 报错解决方案
CUDA Device Query NVIDIA-GeForce-RTX-3060.txt
10-21
Device 0: "NVIDIA GeForce RTX 3060" CUDA Driver Version / Runtime Version 12.1 / 10.0 CUDA Capability Major/Minor version number: 8.6 Total amount of global memory: 12042 MBytes (12626624512 bytes) MapSMtoCores for SM 8.6 is undefined. Default to use 64 Cores/SM MapSMtoCores for SM 8.6 is undefined. Default to use 64 Cores/SM (28) Multiprocessors, ( 64) CUDA Cores/MP: 1792 CUDA Cores GPU Max Clock rate: 1867 MHz (1.87 GHz)
Assertion `idx_dim ＞= 0 && idx_dim ＜ index_size && “index out of bounds“` failed.
最新发布
d8dongdong的博客
03-18 550
完整报错：C:\cb\pytorch_1000000000000\work\aten\src\ATen\native\cuda\ScatterGatherKernel.cu:367: block: [3,0,0], thread: [114,0,0] Assertion。
onnxruntime windows下cuda驱动文件
09-24
onnxruntime windows10下cuda驱动文件
pytorch模型提示超出内存RuntimeError: CUDA out of memory.
01-20
跑模型时出现RuntimeError: CUDA out of memory.错误 查阅了许多相关内容，原因是：GPU显存内存不够 简单总结一下解决方法： 将batch_size改小。 取torch变量标量值时使用item()属性。 可以在测试阶段添加如下代码： with torch.no_grad(): # 停止自动反向计算梯度 参考： https://wisdomai.xyz/tool/pytorch/archives/2301 https://ptorch.com/news/160.html 作者：菜叶儿掉啦
Python RuntimeError: thread.__init__() not called解决方法
09-22
主要介绍了Python RuntimeError: thread.__init__() not called解决方法,需要的朋友可以参考下
解决RuntimeError: CUDA error: device-side assert triggeredCUDA kernel errors...CUDA_LAUNCH_BLOCKING=1
AI小白炼金术师
04-30 6万+
RuntimeError: CUDA error: device-side assert triggeredCUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
解决报错：RuntimeError: CUDA error: device-side assert triggered
qq_50000922的博客
06-25 1万+
如果下方有提：For debugging consider passing CUDA_LAUNCH_BLOCKING=1.我们可以在代码中import os下添加“os.environ[‘CUDA_LAUNCH_BLOCKING’] = “1””，这样可以保证报错信息提示的位置是准确的。我遇到的问题最终显示是在代码的scatter_函数（torch）中存在映射维度不匹配的情况，映射的目标index有0,1两个位置，但被映射对象对应维度上只有1维，因此报错。
RuntimeError: CUDA error: device-side assert triggered 问题的解决思路
热门推荐
Penta_Kill_5的博客
06-21 9万+
问题如下： RuntimeError: CUDA error: device-side assert triggered CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
RuntimeError: CUDA error: device-side assert triggeredCUDA kernel errors might be asynchronously re
m0_74982304的博客
06-16 7829
解决办法：先测试自己的代码有没有代码上的问题，再考虑有没有cuda内存泄漏的问题，但内存泄漏基本上不会发生。在消除这两种可能后，可以在代码最前面加上下面这段代码，这个方法是gpt告诉我的，这也是报错中所提示的，但是并没有解决我的问题，各位老铁不妨一试。我用的是COCO2017数据集，这个数据集比较大，其中图片很多，我在我的dataloader中将所有图片都调成了[3,256,256]的大小。有意思的是，在训练最开始时一切顺利，到了几千张图片后就发生了上述的报错。到此为止跑了十几万张图片都没有再报错。
解决： RuntimeError: CUDA error: device-side assert triggered CUDA kernel errors might be
python_innocent的博客
03-27 4089
主要可能出现的原因：1.传入传出的通道数不对；2.net最终全连接输出的class数目对不上labels；3. devices没有具体到某一个，而是一个list;4. 关了重新再连。建议先用一个单tensor如下，进行检验net的输入输出正确性，再放到dataset上。笔者首先尝试了前三个，结果还是报这个错，确认修改无误了把这玩意关了，再打开，正常运行了。
RuntimeError: Cannot run the event loop while another loop is running(目前没有解决)
12-21
代码如下: import tornado.ioloop import tornado.web from tornado.httpclient import HTTPClient, AsyncHTTPClient class MainHandler(tornado.web.RequestHandler): # 同步 def get(self): ...
Pytorch报错：CUDA error: device-side assert triggered
weixin_37176244的博客
05-17 4254
总是遇到如下的报错，之前一直也没有记录解决的过程，今天记录一下，避免以后再踩坑 pytorch/aten/src/ATen/native/cuda/Loss.cu:106: operator(): block: [0,0,0], thread: [0,0,0] Assertion `input_val >= zero && input_val <= one` failed. /pytorch/aten/src/ATen/native/cuda/Loss.cu:106: opera
【CV】yolov3 RuntimeError: CUDA error: device-side assert triggered
DJames23的博客
07-29 2264
RuntimeError: CUDA error: device-side assert triggered /pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:60: lambda [](int)->auto::operator()(int)->auto: block: [0,0,0], thread: [2,0,0] Assertion `index >= -sizes[i] && index < sizes[i]
Assertion `index ＞= -sizes[i] && index ＜ sizes[i] && “index out of bounds“` failed
qq_36288181的博客
08-16 6982
pytorch报了一堆错， 首先是类似这样的重复出现: /opt/conda/conda-bld/pytorch_1573049304260/work/aten/src/ATen/native/cuda/IndexKernel.cu:60: lambda [](int)->auto::operator()(int)->auto: block: [0,0,0], thread: [32,0,0] Assertion `index >= -sizes[i] && index
[CUDA error记录]CUDA error: device-side assert triggered的一种可能
weixin_41625823的博客
04-27 1万+
[Pytorch error记录]CUDA error: device-side assert triggered的一种可能 具体问题描述分析 在运行Pytorch的过程中，对新建张量的语句frame_volume = torch.zeros((bs, ch, self.ch_num, h, w)).to(x)报了如下错误： RuntimeError: CUDA error: device-side assert triggered 并且有大量类似的CUDA的报错： .../ATen/native/cud
RuntimeError: CUDA error: device-side assert triggered CUDA kernel errors might be asynchronously re
qq_43601378的博客
09-25 2393
CUDA error
【已解决】RuntimeError: CUDA error: device-side assert triggeredCUDA kernel errors might be asynchronous
BetrayFree的博客
11-20 8463
这篇较为详细的记叙了关于一个bug但又不是很麻烦的bug的解决过程，有部分过程没有记录下来，因为感觉对整个理解过程有影响，所以就挑了主要的过程进行记录。
RuntimeError:CUDAerror: device-side assert triggered CUDA kernel errors might be asynchronously rep
11-10
这个错误通常是由于GPU显存不足导致的。解决方法包括： 1. 减小batch_size。 2. 使用更小的模型或减少模型中的参数数量。 3. 将数据类型转换为更小的数据类型，例如从float32转换为float16。 4. 确保在使用GPU时只使用必要的显存，例如在训练时只保留必要的变量。 5. 确保CUDA版本与PyTorch版本兼容。 6. 确保GPU驱动程序已正确安装并更新到最新版本。

“相关推荐”对你有帮助么？

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论 1
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值