mmsegmentation 训练报错 RuntimeError CUDA error an illegal memory access was encountered

File “C:\Users\**\anaconda3\envs\openmmlab1\lib\site-packages\mmengine\runner\loops.py", line 286, in run
self.run_iter(data_batch)
File "C:\Users\**\anaconda3\envs\openmmlab1\lib\site-packages\mmengine\runner\loops.py”, line 309, in run_iter
outputs = self.runner.model.train_step(
File “C:\Users\**\anaconda3\envs\openmmlab1\lib\site-packages\mmengine\model\base_model\base_model.py", line 114, in train_step
losses = self._run_forward(data, mode=‘loss’) # type: ignore
File "C:\Users\**\anaconda3\envs\openmmlab1\lib\site-packages\mmengine\model\base_model\base_model.py”, line 361, in _run_forward
results = self(**data, mode=mode)
File “C:\Users\**\anaconda3\envs\openmmlab1\lib\site-packages\torch\nn\modules\module.py", line 1511, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "C:\Users\**\anaconda3\envs\openmmlab1\lib\site-packages\torch\nn\modules\module.py”, line 1520, in _call_impl
return forward_call(*args, **kwargs)
File “d:\mmsegmentation-1.2.2\mmseg\models\segmentors\base.py”, line 94, in forward
return self.loss(inputs, data_samples)
File “d:\mmsegmentation-1.2.2\mmseg\models\segmentors\encoder_decoder.py”, line 182, in loss
loss_aux = self._auxiliary_head_forward_train(x, data_samples)
File “d:\mmsegmentation-1.2.2\mmseg\models\segmentors\encoder_decoder.py”, line 155, in _auxiliary_head_forward_train
loss_aux = self.auxiliary_head.loss(inputs, data_samples,
File “d:\mmsegmentation-1.2.2\mmseg\models\decode_heads\decode_head.py”, line 262, in loss
losses = self.loss_by_feat(seg_logits, batch_data_samples)
File “d:\mmsegmentation-1.2.2\mmseg\models\decode_heads\decode_head.py”, line 336, in loss_by_feat
loss[‘acc_seg’] = accuracy(
File “d:\mmsegmentation-1.2.2\mmseg\models\losses\accuracy.py”, line 49, in accuracy
correct = correct[:, target != ignore_index]
RuntimeError: CUDA error: an illegal memory access was encountered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.

RuntimeError: CUDA error: an illegal memory access was encountered  
 CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.  
 For debugging consider passing CUDA\_LAUNCH\_BLOCKING=1.  
 Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.


在调试classes的时候碰到的问题，因为隔壁那个classes只有在500/40000到600/40000的时候才会出现，所以用单个powershell跑了好几次，大概三四次之后就开始报这个错，问了AI说很可能是显存不足，之后关了点东西就好了，这个模型刚好卡在我12G显存的边上




---


2024.03.29更新  
 后续在服务器端部署也碰到了这个问题，因为用的是mmsegmentation所以有些网上的方法没法用。但是看到有个方法说是调小batch size能解决，但是我本地用的4080Laptop 12G显存都没问题，我服务器租的4090怎么能出问题呢?

**自我介绍一下，小编13年上海交大毕业，曾经在小公司待过，也去过华为、OPPO等大厂，18年进入阿里一直到现在。**

**深知大多数Python工程师，想要提升技能，往往是自己摸索成长或者是报班学习，但对于培训机构动则几千的学费，着实压力不小。自己不成体系的自学效果低效又漫长，而且极易碰到天花板技术停滞不前！**

**因此收集整理了一份《2024年Python开发全套学习资料》，初衷也很简单，就是希望能够帮助到想自学提升又不知道该从何学起的朋友，同时减轻大家的负担。**

![img](https://img-blog.csdnimg.cn/img_convert/5a97aae93148bcdb0b34e26cf6bd001a.png)

 

![img](https://img-blog.csdnimg.cn/img_convert/6a27d1fe58129d13bf99bcb0289954ff.png)

![img](https://img-blog.csdnimg.cn/img_convert/46506ae54be168b93cf63939786134ca.png)

![img](https://img-blog.csdnimg.cn/img_convert/252731a671c1fb70aad5355a2c5eeff0.png)

![img](https://img-blog.csdnimg.cn/img_convert/6c361282296f86381401c05e862fe4e9.png)

![img](https://img-blog.csdnimg.cn/img_convert/9f49b566129f47b8a67243c1008edf79.png)

 

**既有适合小白学习的零基础资料，也有适合3年以上经验的小伙伴深入学习提升的进阶课程，基本涵盖了95%以上前端开发知识点，真正体系化！**

**由于文件比较大，这里只是将部分目录大纲截图出来，每个节点里面都包含大厂面经、学习笔记、源码讲义、实战项目、讲解视频，并且后续会持续更新**

**如果你觉得这些内容对你有帮助，可以扫码获取！！！（备注Python）**

学习笔记、源码讲义、实战项目、讲解视频，并且后续会持续更新**

**如果你觉得这些内容对你有帮助，可以扫码获取！！！（备注Python）**

<img src="https://img-community.csdnimg.cn/images/fd6ebf0d450a4dbea7428752dc7ffd34.jpg" alt="img" style="zoom:50%;" />