RuntimeError: NCCL error in: /pytorch/torch/lib/c10d/ProcessGroupNCCL.cpp:784 torch

最新推荐文章于 2024-07-09 20:51:12 发布

xzw96

最新推荐文章于 2024-07-09 20:51:12 发布

阅读量7.7k

点赞数 3

分类专栏： pytorch 环境配置

本文链接：https://blog.csdn.net/qq_40630902/article/details/118943104

版权

环境配置同时被 2 个专栏收录

20 篇文章 0 订阅

订阅专栏

pytorch

10 篇文章 0 订阅

订阅专栏

这篇博客讨论了在尝试使用8张GPU运行Detectron2项目时遇到的RuntimeError，错误源于NCCL库的无效使用。问题在于实际GPU数量少于代码配置的数量。解决方案是调整代码以适应可用的两张GPU。博主详细记录了错误日志，并分享了解决多GPU训练问题的经验。

摘要由CSDN通过智能技术生成

-- Process 6 terminated with the following error:
Traceback (most recent call last):
  File "/media/home/intern/anaconda3/envs/torch17/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 19, in _wrap
    fn(i, *args)
  File "/media/home/intern/anaconda3/envs/torch17/lib/python3.7/site-packages/detectron2/engine/launch.py", line 108, in _distributed_worker
    raise e
  File "/media/home/intern/anaconda3/envs/torch17/lib/python3.7/site-packages/detectron2/engine/launch.py", line 103, in _distributed_worker
    timeout=timeout,
  File "/media/home/intern/anaconda3/envs/torch17/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py", line 455, in init_process_group
    barrier()
  File "/media/home/intern/anaconda3/envs/torch17/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py", line 1960, in barrier
    work = _default_pg.barrier()
RuntimeError: NCCL error in: /pytorch/torch/lib/c10d/ProcessGroupNCCL.cpp:784, invalid usage, NCCL version 2.7.8

代码中要求用8张GPU运行，而机器只有两张卡。
所以把代码设置为两张卡运行。