RuntimeError: CUDA error: device-side assert triggered

最新推荐文章于 2024-05-13 10:42:06 发布
最新推荐文章于 2024-05-13 10:42:06 发布
阅读量1k 收藏 1
点赞数
分类专栏： Bug 文章标签： 大数据
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/qq_34845880/article/details/125694989
版权

训练模型的时候出现一下报错：


<torch.utils.data.dataloader.DataLoader object at 0x7f63fe59a588>
Epoch [1/50]
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:108: cunn_ClassNLLCriterion_updateOutput_kernel: block: [0,0,0], thread: [0,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:108: cunn_ClassNLLCriterion_updateOutput_kernel: block: [0,0,0], thread: [2,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:108: cunn_ClassNLLCriterion_updateOutput_kernel: block: [0,0,0], thread: [3,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:108: cunn_ClassNLLCriterion_updateOutput_kernel: block: [0,0,0], thread: [4,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:108: cunn_ClassNLLCriterion_updateOutput_kernel: block: [0,0,0], thread: [5,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:108: cunn_ClassNLLCriterion_updateOutput_kernel: block: [0,0,0], thread: [7,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:108: cunn_ClassNLLCriterion_updateOutput_kernel: block: [0,0,0], thread: [8,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:108: cunn_ClassNLLCriterion_updateOutput_kernel: block: [0,0,0], thread: [9,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:108: cunn_ClassNLLCriterion_updateOutput_kernel: block: [0,0,0], thread: [11,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:108: cunn_ClassNLLCriterion_updateOutput_kernel: block: [0,0,0], thread: [13,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:108: cunn_ClassNLLCriterion_updateOutput_kernel: block: [0,0,0], thread: [14,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:108: cunn_ClassNLLCriterion_updateOutput_kernel: block: [0,0,0], thread: [18,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:108: cunn_ClassNLLCriterion_updateOutput_kernel: block: [0,0,0], thread: [19,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:108: cunn_ClassNLLCriterion_updateOutput_kernel: block: [0,0,0], thread: [20,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:108: cunn_ClassNLLCriterion_updateOutput_kernel: block: [0,0,0], thread: [23,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:108: cunn_ClassNLLCriterion_updateOutput_kernel: block: [0,0,0], thread: [24,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:108: cunn_ClassNLLCriterion_updateOutput_kernel: block: [0,0,0], thread: [27,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:108: cunn_ClassNLLCriterion_updateOutput_kernel: block: [0,0,0], thread: [28,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:108: cunn_ClassNLLCriterion_updateOutput_kernel: block: [0,0,0], thread: [30,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:108: cunn_ClassNLLCriterion_updateOutput_kernel: block: [0,0,0], thread: [31,0,0] Assertion `t >= 0 && t < n_classes` failed.
Traceback (most recent call last):
  File "run.py", line 33, in <module>
    train(model, train_dataloader, val_dataloader, config)
  File "/data/deploy/wang/bertt/bigdata/feedrec/bert_classification/train_eval_pred.py", line 87, in train
    loss, logits, hidden_states = model(batch[0], attention_mask=(batch[0] > 0), labels=batch[1])   # Forward pass
  File "/data/deploy/wang/bertt/lib/python3.6/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/data/deploy/wang/bertt/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 161, in forward
    outputs = self.parallel_apply(replicas, inputs, kwargs)
  File "/data/deploy/wang/bertt/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 171, in parallel_apply
    return parallel_apply(replicas, inputs, kwargs, self.device_ids[:len(replicas)])
  File "/data/deploy/wang/bertt/lib/python3.6/site-packages/torch/nn/parallel/parallel_apply.py", line 86, in parallel_apply
    output.reraise()
  File "/data/deploy/wang/bertt/lib/python3.6/site-packages/torch/_utils.py", line 428, in reraise
    raise self.exc_type(msg)
RuntimeError: Caught RuntimeError in replica 0 on device 0.
Original Traceback (most recent call last):
  File "/data/deploy/wang/bertt/lib/python3.6/site-packages/torch/nn/parallel/parallel_apply.py", line 61, in _worker
    output = module(*input, **kwargs)
  File "/data/deploy/wang/bertt/lib/python3.6/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/data/deploy/wang/bertt/lib/python3.6/site-packages/transformers/modeling_bert.py", line 1344, in forward
    return_dict=return_dict,
  File "/data/deploy/wang/bertt/lib/python3.6/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/data/deploy/wang/bertt/lib/python3.6/site-packages/transformers/modeling_bert.py", line 841, in forward
    return_dict=return_dict,
  File "/data/deploy/wang/bertt/lib/python3.6/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/data/deploy/wang/bertt/lib/python3.6/site-packages/transformers/modeling_bert.py", line 482, in forward
    output_attentions,
  File "/data/deploy/wang/bertt/lib/python3.6/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/data/deploy/wang/bertt/lib/python3.6/site-packages/transformers/modeling_bert.py", line 402, in forward
    output_attentions=output_attentions,
  File "/data/deploy/wang/bertt/lib/python3.6/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/data/deploy/wang/bertt/lib/python3.6/site-packages/transformers/modeling_bert.py", line 339, in forward
    output_attentions,
  File "/data/deploy/wang/bertt/lib/python3.6/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/data/deploy/wang/bertt/lib/python3.6/site-packages/transformers/modeling_bert.py", line 259, in forward
    attention_scores = attention_scores / math.sqrt(self.attention_head_size)
RuntimeError: CUDA error: device-side assert triggered

这是由于标签越界造成的。

Assertion `indexValue >= 0 && indexValue < tensor.sizes[dim]：标签出现了小于0或者大于总数的情况，越界了。调试后发现代码中确实有一部分的标签被人为的-1。当类别为0，再减1就越界了。

其实也很简单
关注
  • 0
    点赞
  • 踩
  • 1
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
专栏目录
完美解决-RuntimeError: CUDA error: device-side assert triggered
jh035的博客
11-23 5813
epoch： 1 的第 0 个inputs torch.Size([8, 3, 224, 224]) labels tensor([4, 4, 3, 4, 4, 4, 4, 4])epoch： 1 的第 4 个inputs torch.Size([8, 3, 224, 224]) labels tensor([3, 4, 2, 4, 1, 1, 4, 1])
RuntimeError: DataLoader worker (pid(s) 9528, 8320) exited unexpectedly
01-06
RuntimeError: DataLoader worker (pid(s) 9528, 8320) exited unexpectedly 去网上搜教程，说是在loader定义的时候多进程的参数指定的问题，我的loader定义如下 loader = Data.DataLoader( dataset=torch_dataset,...
RuntimeError: CUDA error: device-side assert triggered的解决
热门推荐
思念殇千寻的博客
04-14 12万+
　　参考资料：自己debug 　　首先，我报错的问题的文本是：RuntimeError: CUDA error: device-side assert triggered以及 　　Assertion `input_val >= zero && input_val <= one` failed 　　把这两个文本放在前面以便搜索引擎检索。下面说一下我的解决方案，因为问题解...
解决RuntimeError: CUDA error: device-side assert triggeredCUDA kernel errors...CUDA_LAUNCH_BLOCKING=1
AI小白炼金术师
04-30 6万+
RuntimeError: CUDA error: device-side assert triggeredCUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
【已解决】RuntimeError: CUDA error: device-side assert triggeredCUDA kernel errors might be asynchronous
最新发布
dont worry about it的博客
05-13 2万+
报错： RuntimeError: CUDA error: device-side assert triggered CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1. Compile with `TORCH_US
解决报错：RuntimeError: CUDA error: device-side assert triggered
qq_50000922的博客
06-25 1万+
如果下方有提：For debugging consider passing CUDA_LAUNCH_BLOCKING=1.我们可以在代码中import os下添加“os.environ[‘CUDA_LAUNCH_BLOCKING’] = “1””，这样可以保证报错信息提示的位置是准确的。我遇到的问题最终显示是在代码的scatter_函数（torch）中存在映射维度不匹配的情况，映射的目标index有0,1两个位置，但被映射对象对应维度上只有1维，因此报错。
RuntimeError: CUDA error: device-side assert triggered 问题的解决思路
Penta_Kill_5的博客
06-21 9万+
问题如下： RuntimeError: CUDA error: device-side assert triggered CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Python RuntimeError: thread.__init__() not called解决方法
09-22
主要介绍了Python RuntimeError: thread.__init__() not called解决方法,需要的朋友可以参考下
解决pycharm导入numpy包的和使用时报错：RuntimeError: The current Numpy installation (‘D:\\python3.6\\lib\\site-packa的问题
01-21
标题用pycharm导入numpy包的和使用时报错：RuntimeError: The current Numpy installation (‘D:\python3.6\lib\site-packages\numpy\init.py 1.file→settings→project interpreter→+（建议用pychon3.6版本，我...
解决报错RuntimeError: CUDA error: device-side assert triggered
weixin_43787504的博客
10-24 1296
我仔细检查了程序，代码没有问题，后来在调试时发现输入数据中莫名其妙多了一行数据。删除多出的数据后，再运行，程序正常跑起来了。
mmdetection 常见问题及错误记录
01-06
1、GPU下出现/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:53: lambda [](int)->auto::operator()(int)->auto: block: [9,0,0], thread: [32,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && index out of bounds` failed.问题 GPU上运行神经网络模型，是使用多线程并行加速的，如果出错，真正的错误信息很容易被cuda的错误信息淹没，所以难以看到错误信息，一般引起问题的原因是
RuntimeError: CUDA error: device-side assert triggered Compile with `TORCH_USE_CUDA_DSA` to enable
weixin_44913242的博客
09-13 1891
tokenizer.add_special_tokens({‘pad_token’: ‘[PAD]’})添加pad_token的话会导致数据在eos之后还会有值，会导致计算出错。微调baichuan-7b 的时候报了这个错，后面调试了一下发现是因为。baichuan1 的tokenizer类缺少了pad_token。来定义pad_token。
【CV】yolov3 RuntimeError: CUDA error: device-side assert triggered
DJames23的博客
07-29 2310
RuntimeError: CUDA error: device-side assert triggered /pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:60: lambda [](int)->auto::operator()(int)->auto: block: [0,0,0], thread: [2,0,0] Assertion `index >= -sizes[i] && index < sizes[i]
解决：RuntimeError: CUDA error: device-side assert triggered
wotianxiu的博客
04-13 2198
[TOC]解决办法：RuntimeError: CUDA error: device-side assert triggered。
pytorch报错记录
jocker_775065019的博客
03-24 5087
1、RuntimeError: CUDA error: device-side assert triggered 具体报错信息： /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with Inde
runtimeerror: cuda error: device-side assert triggered
04-30
runtimeerror: cuda error: device-side assert triggered 是 CUDA（Compute Unified Device Architecture）在执行GPU计算时发生了错误的提示。在GPU计算过程中，如果程序发现了不符合预期的操作或者输入，就会触发...

“相关推荐”对你有帮助么？

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值