语义分割ADE20K DDP训练Bug

最新推荐文章于 2024-05-07 16:34:11 发布
最新推荐文章于 2024-05-07 16:34:11 发布
阅读量1.7k 收藏 3
点赞数 2
文章标签： bug python 深度学习
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/weixin_41978699/article/details/128223743
版权

在用DDP训练ADE20K数据集时，调试的时候感觉没问题，但是在真的训练的时候报了以下错误

/opt/conda/conda-bld/pytorch_1656352595413/work/aten/src/ATen/native/cuda/NLLLoss2d.cu:103: nll_loss2d_forward_kernel: block: [2,0,0], thread: [864,0,0] Assertion `t >= 0 && t < n_classes` failed.
/opt/conda/conda-bld/pytorch_1656352595413/work/aten/src/ATen/native/cuda/NLLLoss2d.cu:103: nll_loss2d_forward_kernel: block: [2,0,0], thread: [938,0,0] Assertion `t >= 0 && t < n_classes` failed.
/opt/conda/conda-bld/pytorch_1656352595413/work/aten/src/ATen/native/cuda/NLLLoss2d.cu:103: nll_loss2d_forward_kernel: block: [2,0,0], thread: [939,0,0] Assertion `t >= 0 && t < n_classes` failed.
/opt/conda/conda-bld/pytorch_1656352595413/work/aten/src/ATen/native/cuda/NLLLoss2d.cu:103: nll_loss2d_forward_kernel: block: [2,0,0], thread: [940,0,0] Assertion `t >= 0 && t < n_classes` failed.
/opt/conda/conda-bld/pytorch_1656352595413/work/aten/src/ATen/native/cuda/NLLLoss2d.cu:103: nll_loss2d_forward_kernel: block: [2,0,0], thread: [704,0,0] Assertion `t >= 0 && t < n_classes` failed.
/opt/conda/conda-bld/pytorch_1656352595413/work/aten/src/ATen/native/cuda/NLLLoss2d.cu:103: nll_loss2d_forward_kernel: block: [2,0,0], thread: [705,0,0] Assertion `t >= 0 && t < n_classes` failed.
/opt/conda/conda-bld/pytorch_1656352595413/work/aten/src/ATen/native/cuda/NLLLoss2d.cu:103: nll_loss2d_forward_kernel: block: [2,0,0], thread: [706,0,0] Assertion `t >= 0 && t < n_classes` failed.
/opt/conda/conda-bld/pytorch_1656352595413/work/aten/src/ATen/native/cuda/NLLLoss2d.cu:103: nll_loss2d_forward_kernel: block: [2,0,0], thread: [707,0,0] Assertion `t >= 0 && t < n_classes` failed.
/opt/conda/conda-bld/pytorch_1656352595413/work/aten/src/ATen/native/cuda/NLLLoss2d.cu:103: nll_loss2d_forward_kernel: block: [2,0,0], thread: [708,0,0] Assertion `t >= 0 && t < n_classes` failed.
/opt/conda/conda-bld/pytorch_1656352595413/work/aten/src/ATen/native/cuda/NLLLoss2d.cu:103: nll_loss2d_forward_kernel: block: [2,0,0], thread: [709,0,0] Assertion `t >= 0 && t < n_classes` failed.
/opt/conda/conda-bld/pytorch_1656352595413/work/aten/src/ATen/native/cuda/NLLLoss2d.cu:103: nll_loss2d_forward_kernel: block: [2,0,0], thread: [782,0,0] Assertion `t >= 0 && t < n_classes` failed.
/opt/conda/conda-bld/pytorch_1656352595413/work/aten/src/ATen/native/cuda/NLLLoss2d.cu:103: nll_loss2d_forward_kernel: block: [2,0,0], thread: [783,0,0] Assertion `t >= 0 && t < n_classes` failed.
/opt/conda/conda-bld/pytorch_1656352595413/work/aten/src/ATen/native/cuda/NLLLoss2d.cu:103: nll_loss2d_forward_kernel: block: [2,0,0], thread: [784,0,0] Assertion `t >= 0 && t < n_classes` failed.
/opt/conda/conda-bld/pytorch_1656352595413/work/aten/src/ATen/native/cuda/NLLLoss2d.cu:103: nll_loss2d_forward_kernel: block: [2,0,0], thread: [785,0,0] Assertion `t >= 0 && t < n_classes` failed.
/opt/conda/conda-bld/pytorch_1656352595413/work/aten/src/ATen/native/cuda/NLLLoss2d.cu:103: nll_loss2d_forward_kernel: block: [2,0,0], thread: [786,0,0] Assertion `t >= 0 && t < n_classes` failed.
/opt/conda/conda-bld/pytorch_1656352595413/work/aten/src/ATen/native/cuda/NLLLoss2d.cu:103: nll_loss2d_forward_kernel: block: [2,0,0], thread: [465,0,0] Assertion `t >= 0 && t < n_classes` failed.
/opt/conda/conda-bld/pytorch_1656352595413/work/aten/src/ATen/native/cuda/NLLLoss2d.cu:103: nll_loss2d_forward_kernel: block: [2,0,0], thread: [466,0,0] Assertion `t >= 0 && t < n_classes` failed.
/opt/conda/conda-bld/pytorch_1656352595413/work/aten/src/ATen/native/cuda/NLLLoss2d.cu:103: nll_loss2d_forward_kernel: block: [2,0,0], thread: [467,0,0] Assertion `t >= 0 && t < n_classes` failed.
/opt/conda/conda-bld/pytorch_1656352595413/work/aten/src/ATen/native/cuda/NLLLoss2d.cu:103: nll_loss2d_forward_kernel: block: [2,0,0], thread: [1016,0,0] Assertion `t >= 0 && t < n_classes` failed.
/opt/conda/conda-bld/pytorch_1656352595413/work/aten/src/ATen/native/cuda/NLLLoss2d.cu:103: nll_loss2d_forward_kernel: block: [2,0,0], thread: [1017,0,0] Assertion `t >= 0 && t < n_classes` failed.
/opt/conda/conda-bld/pytorch_1656352595413/work/aten/src/ATen/native/cuda/NLLLoss2d.cu:103: nll_loss2d_forward_kernel: block: [2,0,0], thread: [1018,0,0] Assertion `t >= 0 && t < n_classes` failed.
/opt/conda/conda-bld/pytorch_1656352595413/work/aten/src/ATen/native/cuda/NLLLoss2d.cu:103: nll_loss2d_forward_kernel: block: [2,0,0], thread: [860,0,0] Assertion `t >= 0 && t < n_classes` failed.
/opt/conda/conda-bld/pytorch_1656352595413/work/aten/src/ATen/native/cuda/NLLLoss2d.cu:103: nll_loss2d_forward_kernel: block: [2,0,0], thread: [861,0,0] Assertion `t >= 0 && t < n_classes` failed.
/opt/conda/conda-bld/pytorch_1656352595413/work/aten/src/ATen/native/cuda/NLLLoss2d.cu:103: nll_loss2d_forward_kernel: block: [2,0,0], thread: [862,0,0] Assertion `t >= 0 && t < n_classes` failed.
/opt/conda/conda-bld/pytorch_1656352595413/work/aten/src/ATen/native/cuda/NLLLoss2d.cu:103: nll_loss2d_forward_kernel: block: [2,0,0], thread: [863,0,0] Assertion `t >= 0 && t < n_classes` failed.
/opt/conda/conda-bld/pytorch_1656352595413/work/aten/src/ATen/native/cuda/NLLLoss2d.cu:103: nll_loss2d_forward_kernel: block: [2,0,0], thread: [387,0,0] Assertion `t >= 0 && t < n_classes` failed.
/opt/conda/conda-bld/pytorch_1656352595413/work/aten/src/ATen/native/cuda/NLLLoss2d.cu:103: nll_loss2d_forward_kernel: block: [2,0,0], thread: [388,0,0] Assertion `t >= 0 && t < n_classes` failed.
/opt/conda/conda-bld/pytorch_1656352595413/work/aten/src/ATen/native/cuda/NLLLoss2d.cu:103: nll_loss2d_forward_kernel: block: [2,0,0], thread: [389,0,0] Assertion `t >= 0 && t < n_classes` failed.
/opt/conda/conda-bld/pytorch_1656352595413/work/aten/src/ATen/native/cuda/NLLLoss2d.cu:103: nll_loss2d_forward_kernel: block: [2,0,0], thread: [231,0,0] Assertion `t >= 0 && t < n_classes` failed.
/opt/conda/conda-bld/pytorch_1656352595413/work/aten/src/ATen/native/cuda/NLLLoss2d.cu:103: nll_loss2d_forward_kernel: block: [2,0,0], thread: [232,0,0] Assertion `t >= 0 && t < n_classes` failed.
/opt/conda/conda-bld/pytorch_1656352595413/work/aten/src/ATen/native/cuda/NLLLoss2d.cu:103: nll_loss2d_forward_kernel: block: [2,0,0], thread: [233,0,0] Assertion `t >= 0 && t < n_classes` failed.
/opt/conda/conda-bld/pytorch_1656352595413/work/aten/src/ATen/native/cuda/NLLLoss2d.cu:103: nll_loss2d_forward_kernel: block: [2,0,0], thread: [234,0,0] Assertion `t >= 0 && t < n_classes` failed.
/opt/conda/conda-bld/pytorch_1656352595413/work/aten/src/ATen/native/cuda/NLLLoss2d.cu:103: nll_loss2d_forward_kernel: block: [2,0,0], thread: [235,0,0] Assertion `t >= 0 && t < n_classes` failed.
/opt/conda/conda-bld/pytorch_1656352595413/work/aten/src/ATen/native/cuda/NLLLoss2d.cu:103: nll_loss2d_forward_kernel: block: [2,0,0], thread: [236,0,0] Assertion `t >= 0 && t < n_classes` failed.
/opt/conda/conda-bld/pytorch_1656352595413/work/aten/src/ATen/native/cuda/NLLLoss2d.cu:103: nll_loss2d_forward_kernel: block: [2,0,0], thread: [309,0,0] Assertion `t >= 0 && t < n_classes` failed.
/opt/conda/conda-bld/pytorch_1656352595413/work/aten/src/ATen/native/cuda/NLLLoss2d.cu:103: nll_loss2d_forward_kernel: block: [2,0,0], thread: [310,0,0] Assertion `t >= 0 && t < n_classes` failed.
/opt/conda/conda-bld/pytorch_1656352595413/work/aten/src/ATen/native/cuda/NLLLoss2d.cu:103: nll_loss2d_forward_kernel: block: [2,0,0], thread: [311,0,0] Assertion `t >= 0 && t < n_classes` failed.
/opt/conda/conda-bld/pytorch_1656352595413/work/aten/src/ATen/native/cuda/NLLLoss2d.cu:103: nll_loss2d_forward_kernel: block: [2,0,0], thread: [312,0,0] Assertion `t >= 0 && t < n_classes` failed.
/opt/conda/conda-bld/pytorch_1656352595413/work/aten/src/ATen/native/cuda/NLLLoss2d.cu:103: nll_loss2d_forward_kernel: block: [2,0,0], thread: [313,0,0] Assertion `t >= 0 && t < n_classes` failed.
/opt/conda/conda-bld/pytorch_1656352595413/work/aten/src/ATen/native/cuda/NLLLoss2d.cu:103: nll_loss2d_forward_kernel: block: [2,0,0], thread: [548,0,0] Assertion `t >= 0 && t < n_classes` failed.
/opt/conda/conda-bld/pytorch_1656352595413/work/aten/src/ATen/native/cuda/NLLLoss2d.cu:103: nll_loss2d_forward_kernel: block: [2,0,0], thread: [549,0,0] Assertion `t >= 0 && t < n_classes` failed.
/opt/conda/conda-bld/pytorch_1656352595413/work/aten/src/ATen/native/cuda/NLLLoss2d.cu:103: nll_loss2d_forward_kernel: block: [2,0,0], thread: [550,0,0] Assertion `t >= 0 && t < n_classes` failed.
/opt/conda/conda-bld/pytorch_1656352595413/work/aten/src/ATen/native/cuda/NLLLoss2d.cu:103: nll_loss2d_forward_kernel: block: [2,0,0], thread: [551,0,0] Assertion `t >= 0 && t < n_classes` failed.
/opt/conda/conda-bld/pytorch_1656352595413/work/aten/src/ATen/native/cuda/NLLLoss2d.cu:103: nll_loss2d_forward_kernel: block: [2,0,0], thread: [552,0,0] Assertion `t >= 0 && t < n_classes` failed.
/opt/conda/conda-bld/pytorch_1656352595413/work/aten/src/ATen/native/cuda/NLLLoss2d.cu:103: nll_loss2d_forward_kernel: block: [2,0,0], thread: [553,0,0] Assertion `t >= 0 && t < n_classes` failed.
/opt/conda/conda-bld/pytorch_1656352595413/work/aten/src/ATen/native/cuda/NLLLoss2d.cu:103: nll_loss2d_forward_kernel: block: [2,0,0], thread: [626,0,0] Assertion `t >= 0 && t < n_classes` failed.
/opt/conda/conda-bld/pytorch_1656352595413/work/aten/src/ATen/native/cuda/NLLLoss2d.cu:103: nll_loss2d_forward_kernel: block: [2,0,0], thread: [627,0,0] Assertion `t >= 0 && t < n_classes` failed.
/opt/conda/conda-bld/pytorch_1656352595413/work/aten/src/ATen/native/cuda/NLLLoss2d.cu:103: nll_loss2d_forward_kernel: block: [2,0,0], thread: [628,0,0] Assertion `t >= 0 && t < n_classes` failed.
/opt/conda/conda-bld/pytorch_1656352595413/work/aten/src/ATen/native/cuda/NLLLoss2d.cu:103: nll_loss2d_forward_kernel: block: [2,0,0], thread: [629,0,0] Assertion `t >= 0 && t < n_classes` failed.
/opt/conda/conda-bld/pytorch_1656352595413/work/aten/src/ATen/native/cuda/NLLLoss2d.cu:103: nll_loss2d_forward_kernel: block: [2,0,0], thread: [630,0,0] Assertion `t >= 0 && t < n_classes` failed.
/opt/conda/conda-bld/pytorch_1656352595413/work/aten/src/ATen/native/cuda/NLLLoss2d.cu:103: nll_loss2d_forward_kernel: block: [2,0,0], thread: [631,0,0] Assertion `t >= 0 && t < n_classes` failed.
/opt/conda/conda-bld/pytorch_1656352595413/work/aten/src/ATen/native/cuda/NLLLoss2d.cu:103: nll_loss2d_forward_kernel: block: [2,0,0], thread: [320,0,0] Assertion `t >= 0 && t < n_classes` failed.
/opt/conda/conda-bld/pytorch_1656352595413/work/aten/src/ATen/native/cuda/NLLLoss2d.cu:103: nll_loss2d_forward_kernel: block: [2,0,0], thread: [321,0,0] Assertion `t >= 0 && t < n_classes` failed.
/opt/conda/conda-bld/pytorch_1656352595413/work/aten/src/ATen/native/cuda/NLLLoss2d.cu:103: nll_loss2d_forward_kernel: block: [2,0,0], thread: [75,0,0] Assertion `t >= 0 && t < n_classes` failed.
/opt/conda/conda-bld/pytorch_1656352595413/work/aten/src/ATen/native/cuda/NLLLoss2d.cu:103: nll_loss2d_forward_kernel: block: [2,0,0], thread: [76,0,0] Assertion `t >= 0 && t < n_classes` failed.
/opt/conda/conda-bld/pytorch_1656352595413/work/aten/src/ATen/native/cuda/NLLLoss2d.cu:103: nll_loss2d_forward_kernel: block: [2,0,0], thread: [77,0,0] Assertion `t >= 0 && t < n_classes` failed.
/opt/conda/conda-bld/pytorch_1656352595413/work/aten/src/ATen/native/cuda/NLLLoss2d.cu:103: nll_loss2d_forward_kernel: block: [2,0,0], thread: [78,0,0] Assertion `t >= 0 && t < n_classes` failed.
/opt/conda/conda-bld/pytorch_1656352595413/work/aten/src/ATen/native/cuda/NLLLoss2d.cu:103: nll_loss2d_forward_kernel: block: [2,0,0], thread: [79,0,0] Assertion `t >= 0 && t < n_classes` failed.
/opt/conda/conda-bld/pytorch_1656352595413/work/aten/src/ATen/native/cuda/NLLLoss2d.cu:103: nll_loss2d_forward_kernel: block: [2,0,0], thread: [80,0,0] Assertion `t >= 0 && t < n_classes` failed.
/opt/conda/conda-bld/pytorch_1656352595413/work/aten/src/ATen/native/cuda/NLLLoss2d.cu:103: nll_loss2d_forward_kernel: block: [2,0,0], thread: [0,0,0] Assertion `t >= 0 && t < n_classes` failed.
/opt/conda/conda-bld/pytorch_1656352595413/work/aten/src/ATen/native/cuda/NLLLoss2d.cu:103: nll_loss2d_forward_kernel: block: [2,0,0], thread: [1,0,0] Assertion `t >= 0 && t < n_classes` failed.
/opt/conda/conda-bld/pytorch_1656352595413/work/aten/src/ATen/native/cuda/NLLLoss2d.cu:103: nll_loss2d_forward_kernel: block: [2,0,0], thread: [2,0,0] Assertion `t >= 0 && t < n_classes` failed.
/opt/conda/conda-bld/pytorch_1656352595413/work/aten/src/ATen/native/cuda/NLLLoss2d.cu:103: nll_loss2d_forward_kernel: block: [2,0,0], thread: [153,0,0] Assertion `t >= 0 && t < n_classes` failed.
/opt/conda/conda-bld/pytorch_1656352595413/work/aten/src/ATen/native/cuda/NLLLoss2d.cu:103: nll_loss2d_forward_kernel: block: [2,0,0], thread: [154,0,0] Assertion `t >= 0 && t < n_classes` failed.
/opt/conda/conda-bld/pytorch_1656352595413/work/aten/src/ATen/native/cuda/NLLLoss2d.cu:103: nll_loss2d_forward_kernel: block: [2,0,0], thread: [155,0,0] Assertion `t >= 0 && t < n_classes` failed.
/opt/conda/conda-bld/pytorch_1656352595413/work/aten/src/ATen/native/cuda/NLLLoss2d.cu:103: nll_loss2d_forward_kernel: block: [2,0,0], thread: [156,0,0] Assertion `t >= 0 && t < n_classes` failed.
/opt/conda/conda-bld/pytorch_1656352595413/work/aten/src/ATen/native/cuda/NLLLoss2d.cu:103: nll_loss2d_forward_kernel: block: [2,0,0], thread: [157,0,0] Assertion `t >= 0 && t < n_classes` failed.
/opt/conda/conda-bld/pytorch_1656352595413/work/aten/src/ATen/native/cuda/NLLLoss2d.cu:103: nll_loss2d_forward_kernel: block: [2,0,0], thread: [158,0,0] Assertion `t >= 0 && t < n_classes` failed.
/opt/conda/conda-bld/pytorch_1656352595413/work/aten/src/ATen/native/cuda/NLLLoss2d.cu:103: nll_loss2d_forward_kernel: block: [2,0,0], thread: [800,0,0] Assertion `t >= 0 && t < n_classes` failed.
/opt/conda/conda-bld/pytorch_1656352595413/work/aten/src/ATen/native/cuda/NLLLoss2d.cu:103: nll_loss2d_forward_kernel: block: [2,0,0], thread: [801,0,0] Assertion `t >= 0 && t < n_classes` failed.
/opt/conda/conda-bld/pytorch_1656352595413/work/aten/src/ATen/native/cuda/NLLLoss2d.cu:103: nll_loss2d_forward_kernel: block: [2,0,0], thread: [802,0,0] Assertion `t >= 0 && t < n_classes` failed.
/opt/conda/conda-bld/pytorch_1656352595413/work/aten/src/ATen/native/cuda/NLLLoss2d.cu:103: nll_loss2d_forward_kernel: block: [2,0,0], thread: [803,0,0] Assertion `t >= 0 && t < n_classes` failed.
/opt/conda/conda-bld/pytorch_1656352595413/work/aten/src/ATen/native/cuda/NLLLoss2d.cu:103: nll_loss2d_forward_kernel: block: [2,0,0], thread: [160,0,0] Assertion `t >= 0 && t < n_classes` failed.
/opt/conda/conda-bld/pytorch_1656352595413/work/aten/src/ATen/native/cuda/NLLLoss2d.cu:103: nll_loss2d_forward_kernel: block: [2,0,0], thread: [161,0,0] Assertion `t >= 0 && t < n_classes` failed.
/opt/conda/conda-bld/pytorch_1656352595413/work/aten/src/ATen/native/cuda/NLLLoss2d.cu:103: nll_loss2d_forward_kernel: block: [2,0,0], thread: [162,0,0] Assertion `t >= 0 && t < n_classes` failed.
/opt/conda/conda-bld/pytorch_1656352595413/work/aten/src/ATen/native/cuda/NLLLoss2d.cu:103: nll_loss2d_forward_kernel: block: [2,0,0], thread: [163,0,0] Assertion `t >= 0 && t < n_classes` failed.
/opt/conda/conda-bld/pytorch_1656352595413/work/aten/src/ATen/native/cuda/NLLLoss2d.cu:103: nll_loss2d_forward_kernel: block: [2,0,0], thread: [164,0,0] Assertion `t >= 0 && t < n_classes` failed.
/opt/conda/conda-bld/pytorch_1656352595413/work/aten/src/ATen/native/cuda/NLLLoss2d.cu:103: nll_loss2d_forward_kernel: block: [2,0,0], thread: [165,0,0] Assertion `t >= 0 && t < n_classes` failed.
/opt/conda/conda-bld/pytorch_1656352595413/work/aten/src/ATen/native/cuda/NLLLoss2d.cu:103: nll_loss2d_forward_kernel: block: [2,0,0], thread: [166,0,0] Assertion `t >= 0 && t < n_classes` failed.
/opt/conda/conda-bld/pytorch_1656352595413/work/aten/src/ATen/native/cuda/NLLLoss2d.cu:103: nll_loss2d_forward_kernel: block: [2,0,0], thread: [960,0,0] Assertion `t >= 0 && t < n_classes` failed.
/opt/conda/conda-bld/pytorch_1656352595413/work/aten/src/ATen/native/cuda/NLLLoss2d.cu:103: nll_loss2d_forward_kernel: block: [2,0,0], thread: [961,0,0] Assertion `t >= 0 && t < n_classes` failed.
/opt/conda/conda-bld/pytorch_1656352595413/work/aten/src/ATen/native/cuda/NLLLoss2d.cu:103: nll_loss2d_forward_kernel: block: [2,0,0], thread: [962,0,0] Assertion `t >= 0 && t < n_classes` failed.
/opt/conda/conda-bld/pytorch_1656352595413/work/aten/src/ATen/native/cuda/NLLLoss2d.cu:103: nll_loss2d_forward_kernel: block: [2,0,0], thread: [963,0,0] Assertion `t >= 0 && t < n_classes` failed.
/opt/conda/conda-bld/pytorch_1656352595413/work/aten/src/ATen/native/cuda/NLLLoss2d.cu:103: nll_loss2d_forward_kernel: block: [2,0,0], thread: [964,0,0] Assertion `t >= 0 && t < n_classes` failed.
/opt/conda/conda-bld/pytorch_1656352595413/work/aten/src/ATen/native/cuda/NLLLoss2d.cu:103: nll_loss2d_forward_kernel: block: [2,0,0], thread: [640,0,0] Assertion `t >= 0 && t < n_classes` failed.
/opt/conda/conda-bld/pytorch_1656352595413/work/aten/src/ATen/native/cuda/NLLLoss2d.cu:103: nll_loss2d_forward_kernel: block: [2,0,0], thread: [641,0,0] Assertion `t >= 0 && t < n_classes` failed.
/opt/conda/conda-bld/pytorch_1656352595413/work/aten/src/ATen/native/cuda/NLLLoss2d.cu:103: nll_loss2d_forward_kernel: block: [2,0,0], thread: [642,0,0] Assertion `t >= 0 && t < n_classes` failed.
/opt/conda/conda-bld/pytorch_1656352595413/work/aten/src/ATen/native/cuda/NLLLoss2d.cu:103: nll_loss2d_forward_kernel: block: [2,0,0], thread: [643,0,0] Assertion `t >= 0 && t < n_classes` failed.
/opt/conda/conda-bld/pytorch_1656352595413/work/aten/src/ATen/native/cuda/NLLLoss2d.cu:103: nll_loss2d_forward_kernel: block: [2,0,0], thread: [644,0,0] Assertion `t >= 0 && t < n_classes` failed.
/opt/conda/conda-bld/pytorch_1656352595413/work/aten/src/ATen/native/cuda/NLLLoss2d.cu:103: nll_loss2d_forward_kernel: block: [2,0,0], thread: [645,0,0] Assertion `t >= 0 && t < n_classes` failed.
/opt/conda/conda-bld/pytorch_1656352595413/work/aten/src/ATen/native/cuda/NLLLoss2d.cu:103: nll_loss2d_forward_kernel: block: [2,0,0], thread: [646,0,0] Assertion `t >= 0 && t < n_classes` failed.
/opt/conda/conda-bld/pytorch_1656352595413/work/aten/src/ATen/native/cuda/NLLLoss2d.cu:103: nll_loss2d_forward_kernel: block: [2,0,0], thread: [647,0,0] Assertion `t >= 0 && t < n_classes` failed.
/opt/conda/conda-bld/pytorch_1656352595413/work/aten/src/ATen/native/cuda/NLLLoss2d.cu:103: nll_loss2d_forward_kernel: block: [2,0,0], thread: [480,0,0] Assertion `t >= 0 && t < n_classes` failed.
/opt/conda/conda-bld/pytorch_1656352595413/work/aten/src/ATen/native/cuda/NLLLoss2d.cu:103: nll_loss2d_forward_kernel: block: [2,0,0], thread: [481,0,0] Assertion `t >= 0 && t < n_classes` failed.
/opt/conda/conda-bld/pytorch_1656352595413/work/aten/src/ATen/native/cuda/NLLLoss2d.cu:103: nll_loss2d_forward_kernel: block: [2,0,0], thread: [482,0,0] Assertion `t >= 0 && t < n_classes` failed.
/opt/conda/conda-bld/pytorch_1656352595413/work/aten/src/ATen/native/cuda/NLLLoss2d.cu:103: nll_loss2d_forward_kernel: block: [2,0,0], thread: [483,0,0] Assertion `t >= 0 && t < n_classes` failed.
/opt/conda/conda-bld/pytorch_1656352595413/work/aten/src/ATen/native/cuda/NLLLoss2d.cu:103: nll_loss2d_forward_kernel: block: [2,0,0], thread: [484,0,0] Assertion `t >= 0 && t < n_classes` failed.
/opt/conda/conda-bld/pytorch_1656352595413/work/aten/src/ATen/native/cuda/NLLLoss2d.cu:103: nll_loss2d_forward_kernel: block: [2,0,0], thread: [485,0,0] Assertion `t >= 0 && t < n_classes` failed.
/opt/conda/conda-bld/pytorch_1656352595413/work/aten/src/ATen/native/cuda/NLLLoss2d.cu:103: nll_loss2d_forward_kernel: block: [2,0,0], thread: [96,0,0] Assertion `t >= 0 && t < n_classes` failed.
/opt/conda/conda-bld/pytorch_1656352595413/work/aten/src/ATen/native/cuda/NLLLoss2d.cu:103: nll_loss2d_forward_kernel: block: [2,0,0], thread: [256,0,0] Assertion `t >= 0 && t < n_classes` failed.
/opt/conda/conda-bld/pytorch_1656352595413/work/aten/src/ATen/native/cuda/NLLLoss2d.cu:103: nll_loss2d_forward_kernel: block: [2,0,0], thread: [257,0,0] Assertion `t >= 0 && t < n_classes` failed.
terminate called after throwing an instance of 'c10::CUDAError'
  what():  CUDA error: device-side assert triggered
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Exception raised from record at /opt/conda/conda-bld/pytorch_1656352595413/work/aten/src/ATen/cuda/CUDAEvent.h:115 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x4d (0x7fb23875b46d in /home/wangyh/anaconda3/envs/python3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #1: <unknown function> + 0xdd7255 (0x7fb2399d1255 in /home/wangyh/anaconda3/envs/python3.10/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #2: <unknown function> + 0x324198 (0x7fb27cc24198 in /home/wangyh/anaconda3/envs/python3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
frame #3: c10::TensorImpl::release_resources() + 0x175 (0x7fb23873b485 in /home/wangyh/anaconda3/envs/python3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #4: <unknown function> + 0x200dad (0x7fb27cb00dad in /home/wangyh/anaconda3/envs/python3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
frame #5: <unknown function> + 0x52c4b8 (0x7fb27ce2c4b8 in /home/wangyh/anaconda3/envs/python3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
frame #6: THPVariable_subclass_dealloc(_object*) + 0x2b2 (0x7fb27ce2c7b2 in /home/wangyh/anaconda3/envs/python3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
frame #7: <unknown function> + 0x121ebb (0x55798558aebb in /home/wangyh/anaconda3/envs/python3.10/bin/python)
frame #8: <unknown function> + 0x12f3a2 (0x5579855983a2 in /home/wangyh/anaconda3/envs/python3.10/bin/python)
frame #9: <unknown function> + 0x21c50f (0x55798568550f in /home/wangyh/anaconda3/envs/python3.10/bin/python)
frame #10: _PyEval_EvalFrameDefault + 0x579a (0x5579855a226a in /home/wangyh/anaconda3/envs/python3.10/bin/python)
frame #11: _PyFunction_Vectorcall + 0x6f (0x5579855ad5ef in /home/wangyh/anaconda3/envs/python3.10/bin/python)
frame #12: _PyEval_EvalFrameDefault + 0x304 (0x55798559cdd4 in /home/wangyh/anaconda3/envs/python3.10/bin/python)
frame #13: _PyFunction_Vectorcall + 0x6f (0x5579855ad5ef in /home/wangyh/anaconda3/envs/python3.10/bin/python)
frame #14: _PyEval_EvalFrameDefault + 0x304 (0x55798559cdd4 in /home/wangyh/anaconda3/envs/python3.10/bin/python)
frame #15: <unknown function> + 0x1e6a82 (0x55798564fa82 in /home/wangyh/anaconda3/envs/python3.10/bin/python)
frame #16: PyEval_EvalCode + 0x87 (0x55798564f9c7 in /home/wangyh/anaconda3/envs/python3.10/bin/python)
frame #17: <unknown function> + 0x21a399 (0x557985683399 in /home/wangyh/anaconda3/envs/python3.10/bin/python)
frame #18: <unknown function> + 0x214ca4 (0x55798567dca4 in /home/wangyh/anaconda3/envs/python3.10/bin/python)
frame #19: <unknown function> + 0x98028 (0x557985501028 in /home/wangyh/anaconda3/envs/python3.10/bin/python)
frame #20: _PyRun_SimpleFileObject + 0x1af (0x557985677fef in /home/wangyh/anaconda3/envs/python3.10/bin/python)
frame #21: _PyRun_AnyFileObject + 0x43 (0x557985677bb3 in /home/wangyh/anaconda3/envs/python3.10/bin/python)
frame #22: Py_RunMain + 0x39f (0x557985674d5f in /home/wangyh/anaconda3/envs/python3.10/bin/python)
frame #23: Py_BytesMain + 0x39 (0x557985642cf9 in /home/wangyh/anaconda3/envs/python3.10/bin/python)
frame #24: __libc_start_main + 0xe7 (0x7fb2aed07c87 in /lib/x86_64-linux-gnu/libc.so.6)
frame #25: <unknown function> + 0x1d9c01 (0x557985642c01 in /home/wangyh/anaconda3/envs/python3.10/bin/python)

WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 47605 closing signal SIGTERM
WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 47607 closing signal SIGTERM
WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 47609 closing signal SIGTERM
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: -6) local_rank: 2 (pid: 47608) of binary: /home/wangyh/anaconda3/envs/python3.10/bin/python
Traceback (most recent call last):
  File "/home/wangyh/anaconda3/envs/python3.10/bin/torchrun", line 33, in <module>
    sys.exit(load_entry_point('torch==1.12.0', 'console_scripts', 'torchrun')())
  File "/home/wangyh/anaconda3/envs/python3.10/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 345, in wrapper
    return f(*args, **kwargs)
  File "/home/wangyh/anaconda3/envs/python3.10/lib/python3.10/site-packages/torch/distributed/run.py", line 761, in main
    run(args)
  File "/home/wangyh/anaconda3/envs/python3.10/lib/python3.10/site-packages/torch/distributed/run.py", line 752, in run
    elastic_launch(
  File "/home/wangyh/anaconda3/envs/python3.10/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 131, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/home/wangyh/anaconda3/envs/python3.10/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 245, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 
======================================================
train.py FAILED
------------------------------------------------------
Failures:
  <NO_OTHER_FAILURES>
------------------------------------------------------
Root Cause (first observed failure):
[0]:
  time      : 2022-12-07_16:01:32
  host      : Ti-Sev
  rank      : 2 (local_rank: 2)
  exitcode  : -6 (pid: 47608)
  error_file: <N/A>
  traceback : Signal 6 (SIGABRT) received by PID 47608

在网上找了半天，主要问题在于你的实际训练类别和数据集ADE20K的类别不符合。默认的分割数据集会将0设置为背景类，其余的1-150为我们要预测的类别，一般来说设置的背景类是要忽略的

下面看MMCV的一段话

打开加载数据的源码片段可以看到一段处理 reduce_zero_label 的代码，意思是：若开启了 reduce_zero_label，原本为 0 的所有标注设置为 255，也就是损失函数中 ignore_index 参数的默认值，该参数默认避免值为 255 的标注参与损失计算。前文按下不表的 150 类的 ADE 数据集，它不包含背景的原因就是开了 reduce zero label，原本为 0 值的背景设置为了 ignore_index

即，我们在训练时应该将背景0类变为255，然后再optimizer中忽略掉255的分类像素值，这样就不会报错了，代码如下

# mmseg/datasets/pipelines/loading.py

...
# reduce zero_label
if self.reduce_zero_label:
    # avoid using underflow conversion
    gt_semantic_seg[gt_semantic_seg == 0] = 255
    gt_semantic_seg = gt_semantic_seg - 1
    gt_semantic_seg[gt_semantic_seg == 254] = 255
...
harry_tea
关注
  • 2
    点赞
  • 踩
  • 3
    收藏
    觉得还不错? 一键收藏
  • 1
    评论
RuntimeError: CUDA error: device-side assert triggered报错
ruichuang2的博客
07-08 444
在加载数据的时候Dataloader()加上drop_last=True，就好了。这个报错，依次排除了是不是因为数据包含nan；
../aten/src/ATen/native/cuda/NLLLoss2d.cu:103: nll_loss2d_forward_kernel: block: [1,0,0], thread: [4
weixin_42748439的博客
10-11 1334
我的信息比较玄学，不满足以上三个问题中的任何一个，斗智斗勇一天，我的目标全都是指向预测值错了，然后一直没调好。
报错For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
xiao_lxl的专栏
11-23 1170
报错For debugging consider passing CUDA_LAUNCH_BLOCKING=1.问题排查及解决方案
FileNotFoundError: [Errno 2] No such file or directory: ‘data/gym/gym_hrnet.pkl‘
qq_34425255的博客
05-07 807
main()main()run(args)Failures:Failures:[0]:
RuntimeError: CUDA error: device-side assert triggered
m0_63828426的博客
10-27 2667
RuntimeError: CUDA error: device-side assert triggered
Pytorch问题：device-side assert triggered 未必是标签越界导致
热门推荐
qq_40156289的博客
12-26 1万+
详细报错信息如下： 关键错误为： /pytorch/aten/src/ATen/native/cuda/Loss.cu:106: operator(): block: [73,0,0], thread: [30,0,0] Assertion input_val >= zero && input_val <= one failed. THCudaCheck FAIL file=/pytorch/aten/src/THC/THCCachingHostAllocator.cpp lin
写tf和pytorch遇到的一些错误以及解决办法(如果当时解决了的话)
大黄老鼠的小本本
07-17 9222
Traceback (most recent call last): File "/root/userfolder/code/cc/CodeCompletion/CodeCompletionGNN.py", line 569, in <module> GnnTrainer(config).start() File "/root/userfolder/code/cc/CodeCompletion/CodeCompletionGNN.py", line 346, in start
语义分割-基于Pytorch在MIT-ADE20K数据集上实现语义分割+场景理解算法-附项目源码-优质项目实战.zip
最新发布
05-26
基于Pytorch在MIT-ADE20K数据集上实现语义分割+场景理解算法-附项目源码-优质项目实战.zip”的压缩包中，包含了一个全面的语义分割项目，使用了Pytorch深度学习框架，并在MIT-ADE20K数据集上进行了训练和验证。...
MIT ADE20K数据集上用于语义分割/场景解析的Pytorch实现-Python开发
05-25
ADE20K是PyTorch中MIT ADE20K数据集上最大的开源语义分割。这是MIT ADE20K场景解析数据集（http://sceneparsing.csail.mit.edu/）上语义分割模型的PyTorch实现。 ADE20K是由MIT计算机视觉团队发布的最大的用于语义...
ADE20K语义分割色彩参考表格
12-13
用于stable diffusion的control net里的seg模型处理； 让你快速查阅对应的颜色代表的物体是什么，快色编辑修改图片里的色块区域，定制你的专属图片； 尤其适合用于ai室内设计。
在ADE20k语义分割上训练的MaskFormer模型（基本大小的版本，Swin主干）
04-01
在ADE20k语义分割上训练的MaskFormer模型（基本大小的版本，Swin主干）。MaskFormer 使用相同的范式处理实例、语义和全景分割：通过预测一组掩码和相应的标签。因此，所有 3 个任务都被视为实例分段。
ADE20K：ADE20K数据集
03-03
ADE20K数据集 这是的存储库。 我们提供了有关数据集的一些信息，并提供了用于探索数据的。概述ADE20K由来自SUN和Places数据库的27K图像组成。 图像完全用对象注释，跨越3K对象类别。 许多图像还包含对象部分和部分...
Pytorch报错：CUDA error: device-side assert triggered
weixin_37176244的博客
05-17 4383
总是遇到如下的报错，之前一直也没有记录解决的过程，今天记录一下，避免以后再踩坑 pytorch/aten/src/ATen/native/cuda/Loss.cu:106: operator(): block: [0,0,0], thread: [0,0,0] Assertion `input_val >= zero && input_val <= one` failed. /pytorch/aten/src/ATen/native/cuda/Loss.cu:106: opera
分级代码报错调试记录
X922199的博客
09-08 2777
ValueError: Function has keyword-only parameters or annotations, use inspect.signature() API which can support them
RuntimeError: CUDA error: device-side assert triggered CUDA kernel errors might be asynchronously
李白吃肥牛的博客
07-07 5049
针对深度学习出现的一些bug
【YOLOv3】训练新数据集记录
DJames23的博客
11-05 734
参考代码：https://github.com/eriklindernoren/PyTorch-YOLOv3 1.训练时loss出现nan 原因： 默认学习率learning_rate=0.001过大导致梯度爆炸，出现inf与nan值 解决办法： 在yolov3.cfg中降低学习率： 参考博客： 网络训练时出现loss为nan的情况(已解决) 神经网络训练时，出现NaN loss ...
PyTorch报错“/.../Loss.cu: ... [59,0,0] Assertion input_val ＞= zero && input_val ＜= one failed.”
墨门
03-04 2388
1 问题描述 今天在调试代码的时候，出现这样一个错误： PyTorch Assertion: /opt/conda/conda-bld/pytorch_1640811803361/work/aten/src/ATen/native/cuda/Loss.cu:115: operator(): block: [123,0,0], thread: [13,0,0] Assertion input_val >= zero && input_val <= one failed. /opt
Loss.cu:97，RuntimeError: copy_if failed to synchronize: cudaErrorAssert: device-side assert triggere
Jiiny_Li的博客
07-28 1198
服务器：Ubuntu18， 环境：python3.6，pytorch 报错信息 /opt/conda/conda-bld/pytorch_1591914838379/work/aten/src/ATen/native/cuda/Loss.cu:97: operator(): block: [0,0,0], thread: [31,0,0] Assertion input_val >= zero && input_val <= one failed. Traceback (most
在使用cuda（gpu）训练神经网络出错怎么办？调成cpu跑一次看看错在哪了
psp3dcg的博客
08-05 4723
由于在GPU上运行神经网络模型，是使用多线程并行加速的，如果出错，真正的错误信息很容易被cuda的错误信息淹没。所以需要用cpu单线程跑一次，真正的错误信息就会显示出来 eg：在gpu下的错误信息： /opt/conda/conda-bld/pytorch_1556653183467/work/aten/src/ATen/native/cuda/IndexKernel.cu:53: lambd...
街景图像语义分割代码 ADE20K
04-18
ADE20K是一个用于街景图像语义分割的数据集，其中包含了超过20,000张高分辨率的街景图像，涵盖了150个不同的语义类别。为了进行街景图像语义分割任务，可以使用ADE20K数据集提供的代码。 ADE20K数据集提供了一些开源的代码库，可以用于街景图像语义分割任务。以下是一些常用的代码库： 1. DeepLab：DeepLab是一个非常流行的图像语义分割模型，可以用于ADE20K数据集。它基于深度卷积神经网络，并使用空洞卷积和多尺度金字塔池化等技术来提高分割性能。你可以在GitHub上找到DeepLab的开源代码。 2. FCN：全卷积网络（Fully Convolutional Network）是另一个常用的图像语义分割模型，也可以用于ADE20K数据集。FCN通过将全连接层替换为卷积层，实现了端到端的像素级别预测。你可以在GitHub上找到FCN的开源代码。 3. PSPNet：PSPNet（Pyramid Scene Parsing Network）是一种基于金字塔池化的图像语义分割模型，也适用于ADE20K数据集。PSPNet通过在不同尺度上提取特征，并利用金字塔池化来捕捉全局上下文信息，提高了分割性能。你可以在GitHub上找到PSPNet的开源代码。 这些代码库通常提供了预训练的模型和训练脚本，你可以使用它们来进行街景图像语义分割任务。同时，你也可以根据自己的需求进行修改和定制。
评论 1
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值