解决RuntimeError: reduce failed to synchronize: device-side assert triggered问题

最新推荐文章于 2024-05-11 11:02:26 发布
最新推荐文章于 2024-05-11 11:02:26 发布
阅读量4.2k 收藏 5
点赞数 2
分类专栏： pytorch NLP python
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/weixin_43922901/article/details/105359357
版权
NLP 同时被 3 个专栏收录
20 篇文章 5 订阅
订阅专栏
pytorch
15 篇文章 0 订阅
订阅专栏
python
11 篇文章 1 订阅
订阅专栏

首先，上一波报错信息：

/pytorch/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [35,0,0], thread: [96,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [35,0,0], thread: [97,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [35,0,0], thread: [98,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [35,0,0], thread: [99,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [35,0,0], thread: [100,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [35,0,0], thread: [101,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [35,0,0], thread: [102,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [35,0,0], thread: [103,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, 
......
......
......
/pytorch/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [35,0,0], thread: [95,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
Traceback (most recent call last):
  File "../paragrah_selector/para_sigmoid_train.py", line 533, in <module>
    main()
  File "../paragrah_selector/para_sigmoid_train.py", line 463, in main
    eval_loss = eval_model(model, eval_data, device)
  File "../paragrah_selector/para_sigmoid_train.py", line 419, in eval_model
    loss, logits = model(input_ids, segment_ids, input_mask, labels=label_ids)
  File "/home/lisen/.conda/envs/py36/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/lisen/.conda/envs/py36/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 143, in forward
    outputs = self.parallel_apply(replicas, inputs, kwargs)
  File "/home/lisen/.conda/envs/py36/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 153, in parallel_apply
    return parallel_apply(replicas, inputs, kwargs, self.device_ids[:len(replicas)])
  File "/home/lisen/.conda/envs/py36/lib/python3.6/site-packages/torch/nn/parallel/parallel_apply.py", line 83, in parallel_apply
    raise output
  File "/home/lisen/.conda/envs/py36/lib/python3.6/site-packages/torch/nn/parallel/parallel_apply.py", line 59, in _worker
    output = module(*input, **kwargs)
  File "/home/lisen/.conda/envs/py36/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/lisen/caiyun_projects/generative_mrc/paragrah_selector/modeling.py", line 1001, in forward
    loss = loss_fn(logits, labels)
  File "/home/lisen/.conda/envs/py36/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/lisen/.conda/envs/py36/lib/python3.6/site-packages/torch/nn/modules/loss.py", line 504, in forward
    return F.binary_cross_entropy(input, target, weight=self.weight, reduction=self.reduction)
  File "/home/lisen/.conda/envs/py36/lib/python3.6/site-packages/torch/nn/functional.py", line 2027, in binary_cross_entropy
    input, target, weight, reduction_enum)
RuntimeError: reduce failed to synchronize: device-side assert triggered
terminate called after throwing an instance of 'c10::Error'
  what():  CUDA error: device-side assert triggered (insert_events at /pytorch/aten/src/THC/THCCachingAllocator.cpp:470)
frame #0: std::function<std::string ()>::operator()() const + 0x11 (0x7f0e52afc021 in /home/lisen/.conda/envs/py36/lib/python3.6/site-packages/torch/lib/libc10.so)
frame #1: c10::Error::Error(c10::SourceLocation, std::string const&) + 0x2a (0x7f0e52afb8ea in /home/lisen/.conda/envs/py36/lib/python3.6/site-packages/torch/lib/libc10.so)
frame #2: <unknown function> + 0x13dbd92 (0x7f0e5e065d92 in /home/lisen/.conda/envs/py36/lib/python3.6/site-packages/torch/lib/libcaffe2_gpu.so)
frame #3: at::TensorImpl::release_resources() + 0x50 (0x7f0e534c6440 in /home/lisen/.conda/envs/py36/lib/python3.6/site-packages/torch/lib/libcaffe2.so)
frame #4: <unknown function> + 0x2af03b (0x7f0e51bb703b in /home/lisen/.conda/envs/py36/lib/python3.6/site-packages/torch/lib/libtorch.so.1)
frame #5: torch::autograd::Variable::Impl::release_resources() + 0x17 (0x7f0e51e29d27 in /home/lisen/.conda/envs/py36/lib/python3.6/site-packages/torch/lib/libtorch.so.1)
frame #6: <unknown function> + 0x124cfb (0x7f0e8ce4ccfb in /home/lisen/.conda/envs/py36/lib/python3.6/site-packages/torch/lib/libtorch_python.so)
frame #7: <unknown function> + 0x3204af (0x7f0e8d0484af in /home/lisen/.conda/envs/py36/lib/python3.6/site-packages/torch/lib/libtorch_python.so)
frame #8: <unknown function> + 0x3204f1 (0x7f0e8d0484f1 in /home/lisen/.conda/envs/py36/lib/python3.6/site-packages/torch/lib/libtorch_python.so)
<omitting python frames>
frame #25: __libc_start_main + 0xf0 (0x7f0ecf782830 in /lib/x86_64-linux-gnu/libc.so.6)

Aborted (core dumped)
(py36) lisen@octa:~/caiyun_projects/generative_mrc/script$ sh para_sigmoid_train.sh

导致这个现象的原因有几个：
1、labels的标签超出了logits的索引，就是比如logits的-1维为10（索引为0~9），你labels出现了大于9的标签，比如10，11…等等。所以仔细检查下你的labels。
2、你的词向量出问题了，比如位置向量超过了模型预设的长度，词向量超出了词表大小等。

然后，这篇文章的重点，如果只说这两个原因，可能大家还是不太容易找出问题。那么教大家一个简单的debug方法，很明显就知道问题所在。那就是：将模型放到CPU上运行。如果放不下，把batch size调小即可。比如本人调完之后报错如下：

File "../paragrah_selector/para_sigmoid_train.py", line 533, in <module>
    main()
  File "../paragrah_selector/para_sigmoid_train.py", line 463, in main
    eval_loss = eval_model(model, eval_data, device)
  File "../paragrah_selector/para_sigmoid_train.py", line 419, in eval_model
    loss, logits = model(input_ids, segment_ids, input_mask, labels=label_ids)
  File "/home/lisen/.conda/envs/py36/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/lisen/caiyun_projects/generative_mrc/paragrah_selector/modeling.py", line 987, in forward
    _, pooled_output = self.bert(input_ids, token_type_ids, attention_mask, output_all_encoded_layers=False)
  File "/home/lisen/.conda/envs/py36/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/lisen/caiyun_projects/generative_mrc/paragrah_selector/modeling.py", line 705, in forward
    embedding_output = self.embeddings(input_ids, token_type_ids)
  File "/home/lisen/.conda/envs/py36/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/lisen/caiyun_projects/generative_mrc/paragrah_selector/modeling.py", line 281, in forward
    position_embeddings = self.position_embeddings(position_ids)
  File "/home/lisen/.conda/envs/py36/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/lisen/.conda/envs/py36/lib/python3.6/site-packages/torch/nn/modules/sparse.py", line 118, in forward
    self.norm_type, self.scale_grad_by_freq, self.sparse)
  File "/home/lisen/.conda/envs/py36/lib/python3.6/site-packages/torch/nn/functional.py", line 1454, in embedding
    return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
RuntimeError: index out of range at /pytorch/aten/src/TH/generic/THTensorEvenMoreMath.cpp:191

仔细分析可以很明显发现， File “/home/lisen/caiyun_projects/generative_mrc/paragrah_selector/modeling.py”, line 281, in forward
position_embeddings = self.position_embeddings(position_ids) 这里出错了，就是位置向量超出了模型的预设长度值，然后回去检查，发现，果然确实没有将较长的文本截断至该长度，导致出现这个问题。

*Lisen
关注
  • 2
    点赞
  • 踩
  • 5
    收藏
    觉得还不错? 一键收藏
  • 4
    评论
专栏目录
目标检测pytorch报错：RuntimeError: CUDA error: device-side assert triggered
qq_41997274的博客
10-27 5674
目标检测pytorch报错：RuntimeError: CUDA error: device-side assert triggered
RuntimeError: reduce failed to synchronize: device-side assert triggered
foolishpeng的博客
05-06 4221
这是最近在跑一个网络结构的时候报错，具体报错内容如下： ​ C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THCUNN/BCECriterion.cu:42: block: [0,0,0], thread: [3,0,0] Assertion `input >= 0. &...
华为昇腾310B1平台视频解码失败[ERROR] Send frame to vdec failed, errorno:507018
最新发布
cumtchw
05-11 904
华为昇腾310B1平台 [ERROR] Send frame to vdec failed, errorno:507018
Reduce failed to synchronize: device-side assert triggered .
weixin_40939762的博客
12-12 374
https://blog.csdn.net/xiaoxifei/article/details/89328596 网络的输出要映射到0-1区间内，加一个sigmoid函数就好了。
PyTorch: reduce failed to synchronize: device-side assert triggered
weixin_38424903的博客
01-14 401
device-side assert triggered
pytorch：Reduce failed to synchronize: device-side assert triggered .
ncf的博客
08-06 555
在pytorch中，运行模型经常会出现这个错误，这个bug的来源是BCELoss中输入的张量value的范围必须在[0.0,1.0]之类，而有时候模型的输出是超出这个范畴的，因此BCELoss会报错。可以使用下面的解决这个问题：（强行限制到0-1） outputs[outputs < 0.0] = 0.0 outputs[o...
Python RuntimeError: thread.__init__() not called解决方法
12-24
RuntimeError: thread.__init__() not called 复制代码 代码如下: class NotifyTread(threading.Thread):   def __init__(self, params):   threading.Thread.__init__(self) #here。。。。。。   self....
解决pycharm导入numpy包的和使用时报错：RuntimeError: The current Numpy installation (‘D:\\python3.6\\lib\\site-packa的问题
01-21
标题用pycharm导入numpy包的和使用时报错：RuntimeError: The current Numpy installation (‘D:\python3.6\lib\site-packages\numpy\init.py 1.file→settings→project interpreter→+（建议用pychon3.6版本，我...
RuntimeError: Cannot run the event loop while another loop is running(目前没有解决)
12-21
代码如下: import tornado.ioloop import tornado.web from tornado.httpclient import HTTPClient, AsyncHTTPClient class MainHandler(tornado.web.RequestHandler): # 同步 def get(self): ...
RuntimeError: DataLoader worker (pid(s) 9528, 8320) exited unexpectedly
01-06
RuntimeError: DataLoader worker (pid(s) 9528, 8320) exited unexpectedly 去网上搜教程，说是在loader定义的时候多进程的参数指定的问题，我的loader定义如下 loader = Data.DataLoader( dataset=torch_dataset,...
Reduce failed to synchronize: device-side assert triggered .（一个在中文里很难找到解决方案的问题）
xiaoxifei的专栏
04-16 1万+
这里必须吐槽下中文环境的bug搜寻情况，直接输入上面的报错很难找到中文描述对应的解决方案。 这是在使用pytorch 自带的BCELoss所报的错误，在GPU，多GPU与CPU运行loss时都会报这样的错误。 我的pytorch环境是0.4.1 ，低于此版本的pytorch都会报这样的问题，1.0版本以上的没有测试过，不知道。 这个bug的来源是BCELoss中输入的张量value的范围必须...
transform: failed to synchronize: cudaErrorAssert: device-side assert triggered
jst100的博客
07-04 2331
该问题的出现一般会出现损失函数上多半是loss.backward()这一句： 首先是分类问题：   首先检查一下是不是多分类问题的损失函数用成二分类的损失函数，其次检查多分类的损失函数的类别数是不是对应的，然后如果是二分类损失函数binary_cross_entropy（）要记得加上激活函数sigmoid，多分类它们自带的有softmax函数不用管了 然后是回归问题：   对于回归问题，常用的损失函数为MSEloss，而标题中的问题一般不会出现在MSEloss本身，也就是说多半是模型的问题。因此这个时候应当
报错CUDA error: device-side assert triggered
weixin_42535423的博客
07-18 2193
CUDA error: device-side assert triggered /tmp/pip-req-build-ocx5vxk7/aten/src/THCUNN/ClassNLLCriterion.cu:106: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctyp
【RuntimeError: unique_by_key: failed to synchronize】问题解决
kao_lengmian的博客
03-22 369
pytorch训练时，提示错误：RuntimeError: unique_by_key: failed to synchronize: cudaErrorAssert: device-side assert triggered。可以看看config文件中设置的类别数目与数据中实际的类别数目是否一致。问题比较基础，但是报错很多同学不知道是哪里触发的，所以记录一下。
CUDA ERROR: device-side assert triggered at问题的解决
Rhdykx的博客
03-30 8040
最近在用深度学习模型跑自己的数据集时遇到了如标题所示的错误，然后在网上搜索了很多方法，发现是计算dice_loss时出错了，于是检查输入输出以及label标签 用torch.isnan(label_batch).any()和torch.isinf(label_batch).any()函数确定了outputs和label_batch中都没有nan和inf，输出都为False 于是再次根据搜索得到的信息，在运行命令前加了CUDA_LAUNCH_BLOCKING=1，看能不能看到具体问题，然后报错是这样的 /p
CUDA ERROR: device-side assert triggered at 问题及解决思路
热门推荐
qq_31049727的博客
05-29 10万+
cuda error RuntimeError: cuda runtime error (59) : device-side assert triggered at ... 我之前还以为是因为GPU抽风了引发的BUG，所以第一次没有在意，直接又重新开始运行了一次，但是第二次就发现程序在同样的地方断掉了，这也就想起来我以前看到的一个博客，里面有句话的大概意思是这样的： 每次都在同样的地方出错的...
cuda error:device-side assert triggered解决办法
BeckyWan_的博客
05-29 285
可能原因：网络中设定的class的数目与真实标签数不匹配。解决方法：修改class数目。
RuntimeError: reduce failed to synchronize: device-side assert triggered pytorch训练
lwgkzl的博客
12-10 3654
总述： 本意在于记录一下这个问题产生的种种情况，请注意不是显卡坏了，而是代码出错了。 1. nan问题 在训练的时候，长久累积某一个tensor，导致他的值过大产生nan。 还有可能是，某一个参数没有初始化，使得内存地址随意赋值，常出现于nn.paramter()的时候 ...
Assertion `srcIndex ＜ srcSelectDimSize` failed解决方法
cmh0221的博客
03-08 8768
Assertion `srcIndex < srcSelectDimSize` failed解决方法

“相关推荐”对你有帮助么？

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论 4
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值