Ubuntu16.04 安装nccl+openmpi+horovod

1.安装nccl

官网教程：https://docs.nvidia.com/deeplearning/nccl/install-guide/index.html#down

1.Install the repository.
选择local或者network其中一种安装方式
For the local NCCL repository:
sudo dpkg -i nccl-repo-<version>.deb
For the network repository:
sudo dpkg -i nvidia-machine-learning-repo-<version>.deb

2.Update the APT database:
sudo apt update

3.Install the libnccl2 package with APT. Additionally, if you need to compile applications with NCCL, you can install the libnccl-dev package as well:
Note: If you are using the network repository, the following command will upgrade CUDA to the latest version.
sudo apt install libnccl2 libnccl-dev
If you prefer to keep an older version of CUDA, specify a specific version, for example:
sudo apt install libnccl2=2.4.8-1+cuda10.0 libnccl-dev=2.4.8-1+cuda10.0

也可以选择使用nccl源码编译安装：https://github.com/NVIDIA/nccl

1.1在使用 sudo apt update指令时提示以下错误：

Hit:31 http://cn.archive.ubuntu.com/ubuntu xenial-backports InRelease
Err:25 https://developer.download.nvidia.cn/compute/cuda/repos/ubuntu1604/x86_64  InRelease
  Clearsigned file isn't valid, got 'NOSPLIT' (does the network require authentication?)
Ign:27 https://developer.download.nvidia.cn/compute/machine-learning/repos/ubuntu1604/x86_64  InRelease
Hit:32 https://developer.download.nvidia.cn/compute/machine-learning/repos/ubuntu1604/x86_64  Release
Fetched 32 B in 30s (1 B/s)  
Reading package lists... Done
E: Failed to fetch http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/InRelease  Clearsigned file isn't valid, got 'NOSPLIT' (does the network require authentication?)
E: Some index files failed to download. They have been ignored, or old ones used instead.

解决办法：

修改/etc/apt/sources.list.d目录下
cuda.list

原始内容为：
deb http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64 /

修改为：
deb https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64 /

ps： http  修改为https

1.2上述问题解决后 执行sudo apt update  报以下错误

Get:2 http://cn.archive.ubuntu.com/ubuntu xenial-updates/main amd64 linux-image-4.15.0-122-generic amd64 4.15.0-122.124~16.04.1 [7996 kB]
Fetched 21.1 MB in 24s (860 kB/s)                                              
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
 LANGUAGE = "en_HK:en",
 LC_ALL = (unset),
 LANG = "en_HK.UTF-8"
    are supported and installed on your system.
perl: warning: Falling back to the standard locale ("C").
locale: Cannot set LC_ALL to default locale: No such file or directory
(Reading database ... 722980 files and directories currently installed.)
Preparing to unpack .../linux-modules-4.15.0-122-generic_4.15.0-122.124~16.04.1_amd64.deb ...
Unpacking linux-modules-4.15.0-122-generic (4.15.0-122.124~16.04.1) ...
dpkg: error processing archive /var/cache/apt/archives/linux-modules-4.15.0-122-generic_4.15.0-122.124~16.04.1_amd64.deb (--unpack):
 cannot copy extracted data for './boot/config-4.15.0-122-generic' to '/boot/config-4.15.0-122-generic.dpkg-new': failed to write (No space left on device)
No apport report written because the error message indicates a disk full error
                                                                              dpkg-deb: error: subprocess paste was killed by signal (Broken pipe)
Preparing to unpack .../linux-image-4.15.0-122-generic_4.15.0-122.124~16.04.1_amd64.deb ...
Unpacking linux-image-4.15.0-122-generic (4.15.0-122.124~16.04.1) ...
dpkg: error processing archive /var/cache/apt/archives/linux-image-4.15.0-122-generic_4.15.0-122.124~16.04.1_amd64.deb (--unpack):
 cannot copy extracted data for './boot/vmlinuz-4.15.0-122-generic' to '/boot/vmlinuz-4.15.0-122-generic.dpkg-new': failed to write (No space left on device)
No apport report written because the error message indicates a disk full error
                                                                              dpkg-deb: error: subprocess paste was killed by signal (Broken pipe)
Errors were encountered while processing:
 /var/cache/apt/archives/linux-modules-4.15.0-122-generic_4.15.0-122.124~16.04.1_amd64.deb
 /var/cache/apt/archives/linux-image-4.15.0-122-generic_4.15.0-122.124~16.04.1_amd64.deb
E: Sub-process /usr/bin/dpkg returned an error code (1)

参考https://blog.csdn.net/jinguangliu/article/details/78145423

执行修复命令，并重新执行更新和升级，确保完整修复问题。

sudo apt --fix-broken install
sudo apt-get update
sudo apt-get upgrade

执行第一条指令时又报新错误

Get:2 http://cn.archive.ubuntu.com/ubuntu xenial-updates/main amd64 linux-image-4.15.0-122-generic amd64 4.15.0-122.124~16.04.1 [7996 kB]
Fetched 21.1 MB in 24s (860 kB/s)                                              
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
 LANGUAGE = "en_HK:en",
 LC_ALL = (unset),
 LANG = "en_HK.UTF-8"
    are supported and installed on your system.
perl: warning: Falling back to the standard locale ("C").
locale: Cannot set LC_ALL to default locale: No such file or directory
(Reading database ... 722980 files and directories currently installed.)
Preparing to unpack .../linux-modules-4.15.0-122-generic_4.15.0-122.124~16.04.1_amd64.deb ...
Unpacking linux-modules-4.15.0-122-generic (4.15.0-122.124~16.04.1) ...
dpkg: error processing archive /var/cache/apt/archives/linux-modules-4.15.0-122-generic_4.15.0-122.124~16.04.1_amd64.deb (--unpack):
 cannot copy extracted data for './boot/config-4.15.0-122-generic' to '/boot/config-4.15.0-122-generic.dpkg-new': failed to write (No space left on device)
No apport report written because the error message indicates a disk full error
                                                                              dpkg-deb: error: subprocess paste was killed by signal (Broken pipe)
Preparing to unpack .../linux-image-4.15.0-122-generic_4.15.0-122.124~16.04.1_amd64.deb ...
Unpacking linux-image-4.15.0-122-generic (4.15.0-122.124~16.04.1) ...
dpkg: error processing archive /var/cache/apt/archives/linux-image-4.15.0-122-generic_4.15.0-122.124~16.04.1_amd64.deb (--unpack):
 cannot copy extracted data for './boot/vmlinuz-4.15.0-122-generic' to '/boot/vmlinuz-4.15.0-122-generic.dpkg-new': failed to write (No space left on device)
No apport report written because the error message indicates a disk full error
                                                                              dpkg-deb: error: subprocess paste was killed by signal (Broken pipe)
Errors were encountered while processing:
 /var/cache/apt/archives/linux-modules-4.15.0-122-generic_4.15.0-122.124~16.04.1_amd64.deb
 /var/cache/apt/archives/linux-image-4.15.0-122-generic_4.15.0-122.124~16.04.1_amd64.deb
E: Sub-process /usr/bin/dpkg returned an error code (1)

显示failed to write (No space left on device)

df -h 查看 是/dev/sda1 磁盘满了 导致的问题，可以清理以下对应空间，之后问题就解决了

ps:把/boot目录清理一下，手动删除了/boot目录下一些非*4.4.0-178*的文件

 

2.安装openmpi

Open MPI:
到官网下载openmpi-x.x.x.tar.gz
tar -xvzf openmpi-x.x.x.tar.gz
cd openmpi-x.x.x
./configure --prefix="/usr/local/openmpi"
make -j 8
sudo make install
在.bashrc中添加环境变量
export PATH="$PATH:/usr/local/cuda/bin:/usr/local/openmpi/bin"
export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:/usr/local/cuda/lib64:/usr/local/openmpi/lib/"
source ~/.bashrc
打开新的终端使之生效。

3.安装horodvod

源码编译或者直接pip安装：一定要好好看官网教程，一开始也是看各种网上教程，但是都没有成功，最后还是跟着官方教程安装成功。

不过其中也有一个比较重要的原因就是horovod对cuda  mxnet,tf,pytorch各种版本也有一定要求，要注意

此次是在ubuntu18.04 x64-cuda10.0 -pytorch1.6.0-mxnet1.5.0下安装成功

 

需要注意的是，如果服务器上有多个版本的cuda，并且你所使用的并非软链接到/usr/local/cuda的版本，那么需要设置cuda路径等变量：

$ HOROVOD_GPU_ALLREDUCE=NCCL HOROVOD_CUDA_HOME=/usr/local/cuda-9.0 HOROVOD_NCCL_HOME=xxx-nccl-path pip install --no-cache-dir horovod

否则可能出现计算能力不匹配等问题。

https://github.com/horovod/horovod#install