NCCL timeout

AI算法网奇

于 2024-08-01 11:17:59 发布

阅读量247

点赞数

分类专栏： pytorch知识宝典文章标签：深度学习人工智能

本文链接：https://blog.csdn.net/jacke121/article/details/140843081

版权

pytorch知识宝典专栏收录该内容

499 篇文章 227 订阅 ¥29.90 ¥99.00

订阅专栏

超级会员免费看

nccl多卡训练时报异常

ght collective operation timeout: WorkNCCL(SeqNum=1441, OpType=ALLREDUCE, NumelIn=499152341, NumelOut=499152341, Timeout(ms)=600000) ran for 600564 milliseconds before timing out.                                                  
[rank1]:[E ProcessGroupNCCL.cpp:537] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data.                                
[rank1]:[E ProcessGroupNCCL.cpp:543] To avoid data inconsistency, we are taking the entire process down.                                                                                                                             
[rank1]:[E Proce

了解本专栏