Some NCCL operations have failed or timed out.

最新推荐文章于 2025-02-27 10:33:12 发布
最新推荐文章于 2025-02-27 10:33:12 发布
阅读量8.7k 收藏 2
点赞数 5
文章标签： python 深度学习 pytorch
Copyright (c) 2020 Alex. All rights reserved.
本文链接：https://blog.csdn.net/weixin_43336281/article/details/127281361
版权

背景：在两台服务器上通过torchrun进行分布式模型训练。

报错：Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data. To avoid this inconsistency, we are taking the entire process down.

翻译：某些 NCCL 操作失败或超时。由于 CUDA 的异步特性 内核，后续的 GPU 操作可能会在损坏/不完整的数据上运行。为了避免这种不一致，我们正在取消整个过程。

完整报错日志：

Some NCCL operations have failed
 or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/inc
omplete data. To avoid this inconsistency, we are taking the entire process down.
terminate called after throwing an instance of 'std::runtime_error'
  what():  NCCL error: unhandled system error, NCCL version 2.10.3
ncclSystemError: System call (e.g. socket, malloc) or external library call failed or device error. It can be also 
caused by unexpected exit of a remote peer, you can check NCCL warnings for failure reason and see if there is conn
ection closure by a peer.
Fatal Python error: Aborted

Thread 0x00007ff9b3476700 (most recent call first):
<no Python frame>

Thread 0x00007ff9b2c75700 (most recent call first):
<no Python frame>

Thread 0x00007ff9b3c77700 (most recent call first):
<no Python frame>

Thread 0x00007ff9b547a700 (most recent call first):
<no Python frame>

Thread 0x00007ff9b4478700 (most recent call first):
<no Python frame>

Thread 0x00007ff9b5c7b700 (most recent call first):
<no Python frame>

Thread 0x00007ff9b4c79700 (most recent call first):
<no Python frame>

Thread 0x00007ff9b67fc700 (most recent call first):
<no Python frame>

Thread 0x00007ff9b7fff700 (most recent call first):
  File "/home/liuzhaofeng/anaconda3/lib/python3.9/threading.py", line 316 in wait
  File "/home/liuzhaofeng/anaconda3/lib/python3.9/queue.py", line 180 in get
  File "/home/liuzhaofeng/anaconda3/lib/python3.9/site-packages/tensorboard/summary/writer/event_file_writer.py", l
ine 227 in run
  File "/home/liuzhaofeng/anaconda3/lib/python3.9/threading.py", line 973 in _bootstrap_inner
  File "/home/liuzhaofeng/anaconda3/lib/python3.9/threading.py", line 930 in _bootstrap

Thread 0x00007ff9d3ae2700 (most recent call first):
  File "/home/liuzhaofeng/anaconda3/lib/python3.9/threading.py", line 316 in wait
  File "/home/liuzhaofeng/anaconda3/lib/python3.9/threading.py", line 574 in wait
  File "/home/liuzhaofeng/anaconda3/lib/python3.9/site-packages/tqdm/_monitor.py", line 60 in run
  File "/home/liuzhaofeng/anaconda3/lib/python3.9/threading.py", line 973 in _bootstrap_inner
  File "/home/liuzhaofeng/anaconda3/lib/python3.9/threading.py", line 930 in _bootstrap

Thread 0x00007ffa68d63180 (most recent call first):
  File "/home/liuzhaofeng/anaconda3/lib/python3.9/site-packages/torch/distributed/distributed_c10d.py", line 2070 i
n all_gather
  File "/home/liuzhaofeng/anaconda3/lib/python3.9/site-packages/deepspeed/comm/torch.py", line 79 in all_gather    
  File "/home/liuzhaofeng/anaconda3/lib/python3.9/site-packages/deepspeed/comm/comm.py", line 152 in all_gather    
  File "/home/liuzhaofeng/anaconda3/lib/python3.9/site-packages/deepspeed/runtime/utils.py", line 1016 in all_gathe
r_dp_groups
  File "/home/liuzhaofeng/anaconda3/lib/python3.9/site-packages/deepspeed/runtime/zero/stage_1_and_2.py", line 1805
 in step
  File "/home/liuzhaofeng/anaconda3/lib/python3.9/site-packages/deepspeed/runtime/engine.py", line 1814 in _take_mo
del_step
  File "/home/liuzhaofeng/anaconda3/lib/python3.9/site-packages/deepspeed/runtime/engine.py", line 1913 in step    
  File "/home/liuzhaofeng/anaconda3/lib/python3.9/site-packages/transformers/trainer.py", line 1756 in _inner_train
ing_loop
  File "/home/liuzhaofeng/anaconda3/lib/python3.9/site-packages/transformers/trainer.py", line 1498 in train       
  File "/home/liuzhaofeng/nlg_pipeline/gpt2/dialog/finetune_gpt2.py", line 126 in train
  File "/home/liuzhaofeng/anaconda3/lib/python3.9/site-packages/torch/distributed/elastic/multiprocessing/errors/__
init__.py", line 345 in wrapper
  File "/home/liuzhaofeng/nlg_pipeline/gpt2/dialog/finetune_gpt2.py", line 132 in <module>
[E ProcessGroupNCCL.cpp:414] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA 
kernels, subsequent GPU operations might run on corrupted/incomplete data. To avoid this inconsistency, we are taki
ng the entire process down.
terminate called after throwing an instance of 'std::runtime_error'
  what():  NCCL error: unhandled system error, NCCL version 2.10.3
ncclSystemError: System call (e.g. socket, malloc) or external library call failed or device error. It can be also 
caused by unexpected exit of a remote peer, you can check NCCL warnings for failure reason and see if there is conn
ection closure by a peer.
Fatal Python error: Aborted

Thread 0x00007f75b7fff700 (most recent call first):
<no Python frame>

Thread 0x00007f75c4ffd700 (most recent call first):
<no Python frame>

Thread 0x00007f75c57fe700 (most recent call first):
<no Python frame>

Thread 0x00007f75c5fff700 (most recent call first):
<no Python frame>

Thread 0x00007f75def7d700 (most recent call first):
<no Python frame>

Thread 0x00007f75df77e700 (most recent call first):
<no Python frame>

Thread 0x00007f75dff7f700 (most recent call first):
<no Python frame>

Thread 0x00007f72f37fe700 (most recent call first):
<no Python frame>

Thread 0x00007f7655632700 (most recent call first):
  File "/home/liuzhaofeng/anaconda3/lib/python3.9/threading.py", line 316 in wait
  File "/home/liuzhaofeng/anaconda3/lib/python3.9/threading.py", line 574 in wait
  File "/home/liuzhaofeng/anaconda3/lib/python3.9/site-packages/tqdm/_monitor.py", line 60 in run
  File "/home/liuzhaofeng/anaconda3/lib/python3.9/threading.py", line 973 in _bootstrap_inner
  File "/home/liuzhaofeng/anaconda3/lib/python3.9/threading.py", line 930 in _bootstrap

Thread 0x00007f76ea8b2180 (most recent call first):
  File "/home/liuzhaofeng/anaconda3/lib/python3.9/site-packages/torch/cuda/__init__.py", line 496 in synchronize   
  File "/home/liuzhaofeng/anaconda3/lib/python3.9/site-packages/deepspeed/utils/timer.py", line 189 in stop        
  File "/home/liuzhaofeng/anaconda3/lib/python3.9/site-packages/deepspeed/runtime/engine.py", line 1915 in step    
  File "/home/liuzhaofeng/anaconda3/lib/python3.9/site-packages/transformers/trainer.py", line 1756 in _inner_train
ing_loop
  File "/home/liuzhaofeng/anaconda3/lib/python3.9/site-packages/transformers/trainer.py", line 1498 in train       
  File "/home/liuzhaofeng/nlg_pipeline/gpt2/dialog/finetune_gpt2.py", line 126 in train
  File "/home/liuzhaofeng/anaconda3/lib/python3.9/site-packages/torch/distributed/elastic/multiprocessing/errors/__
init__.py", line 345 in wrapper
  File "/home/liuzhaofeng/nlg_pipeline/gpt2/dialog/finetune_gpt2.py", line 132 in <module>
WARNING:torch.distributed.elastic.agent.server.api:Received 1 death signal, shutting down workers
WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 4120955 closing signal SIGHUP
WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 4120956 closing signal SIGHUP
WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 4120957 closing signal SIGHUP
WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 4120958 closing signal SIGHUP
WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 4120959 closing signal SIGHUP
WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 4120960 closing signal SIGHUP
WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 4120961 closing signal SIGHUP
WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 4120962 closing signal SIGHUP
Traceback (most recent call last):
  File "/home/liuzhaofeng/anaconda3/bin/torchrun", line 33, in <module>
    sys.exit(load_entry_point('torch==1.12.1', 'console_scripts', 'torchrun')())
  File "/home/liuzhaofeng/anaconda3/lib/python3.9/site-packages/torch/distributed/elastic/multiprocessing/errors/__
init__.py", line 345, in wrapper
    return f(*args, **kwargs)
  File "/home/liuzhaofeng/anaconda3/lib/python3.9/site-packages/torch/distributed/run.py", line 761, in main       
    run(args)
  File "/home/liuzhaofeng/anaconda3/lib/python3.9/site-packages/torch/distributed/run.py", line 752, in run        
    elastic_launch(
  File "/home/liuzhaofeng/anaconda3/lib/python3.9/site-packages/torch/distributed/launcher/api.py", line 131, in __
call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/home/liuzhaofeng/anaconda3/lib/python3.9/site-packages/torch/distributed/launcher/api.py", line 236, in la
unch_agent
    result = agent.run()
  File "/home/liuzhaofeng/anaconda3/lib/python3.9/site-packages/torch/distributed/elastic/metrics/api.py", line 125
, in wrapper
    result = f(*args, **kwargs)
  File "/home/liuzhaofeng/anaconda3/lib/python3.9/site-packages/torch/distributed/elastic/agent/server/api.py", lin
e 709, in run
    result = self._invoke_run(role)
  File "/home/liuzhaofeng/anaconda3/lib/python3.9/site-packages/torch/distributed/elastic/agent/server/api.py", lin
e 850, in _invoke_run
    time.sleep(monitor_interval)
  File "/home/liuzhaofeng/anaconda3/lib/python3.9/site-packages/torch/distributed/elastic/multiprocessing/api.py", 
line 60, in _terminate_process_handler
    raise SignalException(f"Process {os.getpid()} got signal: {sigval}", sigval=sigval)
torch.distributed.elastic.multiprocessing.api.SignalException: Process 4120889 got signal: 1

重点：

Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data. To avoid this inconsistency, we are taking the entire process down.
terminate called after throwing an instance of 'std::runtime_error'
  what():  NCCL error: unhandled system error, NCCL version 2.10.3
ncclSystemError: System call (e.g. socket, malloc) or external library call failed or device error. It can be also caused by unexpected exit of a remote peer, you can check NCCL warnings for failure reason and see if there is connection closure by a peer.

翻译：

某些 NCCL 操作失败或超时。由于 CUDA 的异步特性
内核，后续的 GPU 操作可能会在损坏/不完整的数据上运行。为了避免这种不一致，我们正在取消整个过程。
抛出“std:：runtime_error”实例后调用terminate
what（）：NCCL错误：未处理的系统错误，NCCL版本2.10.3
ncclSystemError：系统调用（例如socket、malloc）或外部库调用失败或设备错误。这也可能是由远程对等端意外退出引起的，您可以检查NCCL警告以了解失败原因，并查看对等端是否关闭了连接。

看样子貌似是两台机器没有同步导致的报错，并且这个问题也是偶发性的，可以先重启一下看看能不能解决。

NCCL timeout
jacke121的专栏
08-01 880
NCCL timeout
大模型微调出错的解决方案（持续更新）
热门推荐
herosunly的博客
06-13 19万+
本文主要介绍了大模型微调出错的解决方案（持续更新），希望对学习大语言模型的同学们有所帮助。 文章目录 1. 前言 2. 解决方案 2.1 大模型环境配置问题 2.2 实时观测GPU变化 2.3 微调卡住不动
Pytorch 多机多卡报错：Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA
flyingluohaipeng的博客
11-17 3762
Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data. To avoid this inconsistency, we are taking the entire process down.多台机器上面的CUDA和Pytorch版本号不一致，必须
解决NCCL网络连接报错
清水渔渔的博客
02-27 917
【代码】解决NCCL网络连接报错。
Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subse
qq_49874815的博客
04-02 4858
2、第二步是看是不是某一张卡出现了问题，过程是分别设置0、1卡，0、1、2、3卡，0、1、2、3、4、5卡，0、1、2、3、4、5、6卡，我的实验中这些都没问题，但是0、1、2、3、4、5、6、7就有问题了，所以怀疑是7卡有问题，然后试了一下1、2、3、4、5、6、7卡跑，也没问题，所以怀疑是0卡和7卡之间有冲突，然后试了一下0、7卡，也没有问题。所以问题不是在某一张卡上。1、首先先看1张卡能不能跑通，以此来排除代码本身问题。代码运行成功，问题解决。
Some NCCL operations have failed or timed out. Due to the asynchronous natur
jacke121的专栏
07-19 1305
Some NCCL operations have failed or timed out. Due to the asynchronous natur
DDP训练Pytorch模型，训练卡死，NCCL通信失败
weixin_45095712的博客
12-07 1130
DDP训练Pytorch模型，训练卡死，NCCL通信失败
[E ProcessGroupNCCL.cpp:737]和[E ProcessGroupNCCL.cpp:414] 解决
shysea2019的博客
01-17 1869
禁用P2P通信：通过设置NCCL_P2P_DISABLE=1，你实际上告诉NCCL不要尝试使用直接的GPU-GPU通信，而是使用其他通信方式（例如通过CPU或网络）。理想的情况是深入了解为何P2P通信在你的特定配置中存在问题，并找到一个更根本的解决方案。P2P通信问题：NCCL利用GPU之间的直接通信（如果可用）来加速操作。然而，在某些系统上，这种直接通信可能会有问题，特别是在复杂的多节点或多卡设置中。网络拓扑和硬件兼容性：在不同的硬件和网络配置下，P2P通信的性能和稳定性可能会有很大差异。
mmdet3d+waymo 踩坑+验证环境正确性流程
ZLTJohn的博客
06-06 1万+
前面配mmdet3d的时候，由于使用了最新版mmdet3d v1.0.0rc2，导致使用官方的config和model，nuscenes数据集上的eval和train结果都不对，后面用了同学环境的版本才好了，但这个时候测waymo就会报错，找了很久bug，才发现新版cuda，旧版torch和tensorflow存在一定程度的冲突，以至于一起用显卡的时候会出现问题。有空交个issue。......
解决 NCCL 通信超时问题：NCCL_BLOCKING_WAIT 的使用与优化
q742971636的博客
10-24 2428
使用是解决 NCCL 超时问题的一个有效方法，它让分布式训练过程中的通信操作更加稳定，尤其是在网络条件不理想或通信数据量较大的情况下。尽管这并不能直接提高通信性能，但它有效地避免了超时导致的训练中断问题。如果在分布式训练中遇到类似的通信超时问题，可以尝试开启阻塞等待模式，并结合其他优化措施，来提高训练的整体稳定性和效率。
RuntimeError: NCCL error in: ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1123, internal erro
qq_49874815的博客
04-03 2628
用命令ifconfig查看网卡，然后重新设置。
Pytorch - 弹性训练原理
whaosoft143ai的博客
08-12 8999
Pytorch在1.9.0引入了torchrun，用其替代1.9.0以前版本的。torchrun在功能的基础上主要新增了两个功能：Failover: 当worker训练失败时，会自动重新启动所有worker继续进行训练；Elastic: 可以动态增加或或删除node节点，本文将通过一个例子说明Elastic Training应该如何使用；...
docker下运行分布式代码报nccl错误：connect returned Connection timed out，成功解决
weixin_45666880的博客
02-22 925
原文：相关内容参考：docker 容器内安装 nccl 后，测试是否安装成功：使用 NVIDIA公司官方提供的测试工具：下载后，进行编译： make默认，cuda的安装位置：/usr/local/cuda默认，nccl的安装位置：/usr如果cuda和nccl的安装位置不是默认的，而是手动选择其他地方的，那么需要在make的时候配置路径：CUDA_HOME 为 cuda的安装路径。NCCL_HOME 为 nccl的安装路径。
[E ProcessGroupNCCL.cpp:828] [Rank 3] Watchdog caught collective operation timeout: WorkNCCL...
bulucc的博客
01-09 7524
Watchdog caught collective operation timeout
deepspeed 训练多机多卡报错 ncclSystemError Last error
高颜值的杀生丸(此博客转载自我的博客园)
04-18 1745
最近在搞分布式训练大模型，踩了两个晚上的坑今天终于爬出来了 我们使用 2台 8*H100 遇到过 错误1 10.255.19.85: ncclSystemError: System call (e.g. socket, malloc) or external library call failed or device error. 10.255.19.85: Last error:10.255....
NVIDIA NCCL 源码学习（六）- channel搜索
KIDGIN7439的专栏
12-01 9035
nccl中channel的概念表示一个通信路径，为了更好的利用带宽和网卡，以及同一块数据可以通过多个channel并发通信，另外后续可以看到一个channel对应了一个GPU SM，所以基于这些原因，nccl会使用多channel，搜索的过程就是搜索出来一组channel。
pytorch中DistributedDataParallel的基本使用框架
水蓝城
01-30 881
Pytorch DistributedDataParallel基本使用
NCCL 故障排除 一
s_sunnyy的博客
01-10 1万+
官方文档 http://docs.nvidia.com/deeplearning/sdk/nccl-developer-guide/index.html#troubleshooting ======================================================================== 5. Troubleshooting  NCCL 故障排
NCCL 故障排除 二
s_sunnyy的博客
01-10 8685
5.4. NCCL Knobs A knob isa type of environment variable that can you can turn on or off by settingspecific values. These environment variables should be set in the context ofrunning NCCL. The followi
Warning: WARNING: process group has NOT been destroyed before we destruct ProcessGroupNCCL. On normal program exit, the application should call destroy_process_group to ensure that any pending NCCL operations have finished in this process. In rare cases this process can exit before this point and block the progress of another member of the process group. This constraint has always been present, but this warning has only been added since PyTorch 2.4 (function operator())什么错误
最新发布
03-13
### PyTorch 中 ProcessGroupNCCL 未销毁导致的警告问题分析 在分布式训练场景下，PyTorch 使用 NCCL 后端来实现高效的 GPU 数据通信。当程序结束时，如果 `ProcessGroupNCCL` 对象未能被正确销毁，则可能会触发如下警告： > "Warning: ProcessGroupNCCL is not destroyed at the end of distributed job." 此警告表明，在进程退出前，某些资源尚未释放完毕，可能导致潜在的内存泄漏或其他不可预期的行为。 #### 警告原因解析 该问题的根本原因是分布式环境中的资源管理不当。具体来说： - 当使用 `torch.distributed.init_process_group(backend='nccl')` 初始化分布式环境后，会创建一个名为 `ProcessGroupNCCL` 的对象。 - 如果在程序正常退出之前没有显式调用 `destroy_process_group()` 或者通过其他方式清理这些资源，则会在 Python 解释器关闭期间尝试自动销毁它们[^1]。 - 这种延迟销毁行为可能与其他全局变量（如单例模式下的实例）发生冲突，尤其是在多线程或多设备环境中。 --- ### 解决方案 以下是几种常见的解决方案及其适用场景： #### 方法一：手动销毁过程组 可以通过显式调用 `dist.destroy_process_group()` 来确保所有与分布相关的资源都被及时释放。例如： ```python import torch.distributed as dist def cleanup(): """Destroy process group to avoid warnings.""" dist.destroy_process_group() ``` 在主脚本中，可以将上述方法作为最后一步执行： ```python if __name__ == "__main__": # Initialize and run your training loop here... # Cleanup after finishing all tasks. cleanup() ``` 这种方法简单有效，适用于大多数情况，并能显著减少因资源未释放而引发的问题。 --- #### 方法二：调整析构函数逻辑 对于更复杂的项目结构，尤其是涉及自定义类封装的情况下，可以在相关类的析构函数中加入额外的安全措施。然而需要注意的是，正如引用所提到的内容那样，“不建议在析构函数中调用 aclFinalize 接口”，因为这可能导致由于单例析构顺序不确定带来的崩溃风险。 因此，推荐的做法是在适当的时间点主动调用必要的清理操作而不是依赖于默认的析构机制。 --- #### 方法三：设置正确的设备上下文 另一个常见问题是多个设备间的切换混乱也可能间接影响到最终状态处理阶段的表现。为此，在初始化以及后续每次更改目标 device ID 前应始终确认当前使用的 device 设置无误: ```cpp // Example C++ equivalent pseudo-code demonstrating setting a specific device before operations. aclError err = aclrtSetDevice(deviceId); if (err != ACL_SUCCESS) { std::cerr << "Failed to set device!" << std::endl; } ``` 虽然这是基于 ACL API 的例子,但在 PyTorch 应用里同样重要——即保证在整个生命周期内只针对单一明确指定的 GPU 设备工作除非确实有必要跨不同卡之间转移数据流. --- ### 总结代码片段 综合以上讨论，下面给出一段完整的示范代码用于消除此类警告消息： ```python import os import torch import torch.distributed as dist def setup(rank, world_size): os.environ['MASTER_ADDR'] = 'localhost' os.environ['MASTER_PORT'] = '12355' # initialize the process group dist.init_process_group( backend='nccl', init_method='env://', world_size=world_size, rank=rank) def cleanup(): dist.destroy_process_group() def main(): rank = ... # Define or get from environment variables etc. size = ... # Similarly define total number of processes setup(rank, size) try: # Your actual computation goes here pass finally: cleanup() # Ensure this runs even on exceptions! if __name__ == '__main__': main() ``` --- ###
评论 1
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包

打赏作者

Matrix 工作室

你的鼓励将是我创作的最大动力

¥1 ¥2 ¥4 ¥6 ¥10 ¥20
扫码支付：¥1
获取中
扫码支付

您的余额不足，请更换扫码支付或充值

打赏作者

实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值