`NCCL_P2P_DISABLE=“1“` and `NCCL_IB_DISABLE=“1“ 及NCCL 安装方法

修改环境变量看是否能解决问题

使用python 代码前写入

# python代码中添加
import os

os.environ["NCCL_P2P_DISABLE"] = "1"
os.environ["NCCL_IB_DISABLE"] = "1"

or bash写入

echo 'export NCCL_P2P_DISABLE=1' >> ~/.bashrc
echo 'export NCCL_IB_DISABLE=1' >> ~/.bashrc
source ~/.bashrc

发现没有效果，检查是否安装nccl

import torch
#单卡
x = torch.rand(1024, 1024, device='cuda:0')
print(torch.cuda.nccl.is_available(x))
#多卡
x = torch.rand(1024, 1024, device='cuda:0')
y = torch.rand(1024, 1024, device='cuda:1')
print(torch.cuda.nccl.is_available([x, y]))

安装方法：

Download NCCL 2.21.5, for CUDA 12.2, April 3rd, 2024

Local installers (x86)

• O/S agnostic local installer
• Local installer for Ubuntu 20.04
• Local installer for RedHat/CentOS 7

Network installers (x86)

If not already done, make sure you install the network repository:

Network Installer for Ubuntu20.04

• $ wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/cuda-keyring_1.0-1_all.deb
• $ sudo dpkg -i cuda-keyring_1.0-1_all.deb
• $ sudo apt-get update

Network Installer for RedHat/CentOS 7

• $ sudo yum-config-manager --add-repo https://developer.download.nvidia.com/compute/cuda/repos/rhel7/x86_64/cuda-rhel7.repo

then run the following command to installer NCCL:
For Ubuntu: sudo apt install libnccl2=2.21.5-1+cuda12.2 libnccl-dev=2.21.5-1+cuda12.2
For RHEL/Centos: sudo yum install libnccl-2.21.5-1+cuda12.2 libnccl-devel-2.21.5-1+cuda12.2 libnccl-static-2.21.5-1+cuda12.2