1.安装nccl
官网教程:https://docs.nvidia.com/deeplearning/nccl/install-guide/index.html#down
1.Install the repository.
选择local或者network其中一种安装方式
For the local NCCL repository:
sudo dpkg -i nccl-repo-<version>.deb
For the network repository:
sudo dpkg -i nvidia-machine-learning-repo-<version>.deb
2.Update the APT database:
sudo apt update
3.Install the libnccl2 package with APT. Additionally, if you need to compile applications with NCCL, you can install the libnccl-dev package as well:
Note: If you are using the network repository, the following command will upgrade CUDA to the latest version.
sudo apt install libnccl2 libnccl-dev
If you prefer to keep an older version of CUDA, specify a specific version, for example:
sudo apt install libnccl2=2.4.8-1+cuda10.0 libnccl-dev=2.4.8-1+cuda10.0
也可以选择使用nccl源码编译安装:https://github.com/NVIDIA/nccl
1.1在使用 sudo apt update指令时提示以下错误:
Hit:31 http://cn.archive.ubuntu.com/ubuntu xenial-backports InRelease
Err:25 https://developer.download.nvidia.cn/compute/cuda/repos/ubuntu1604/x86_64 InRelease
Clearsigned file isn't valid, got 'NOSPLIT' (does the network require authentication?)
Ign:27 https://developer.download.nvidia.cn/compute/machine-learning/repos/ubuntu1604/x86_64 InRelease
Hit:32 https://developer.download.nvidia.cn/compute/machine-learning/repos/ubuntu1604/x86_64 Release
Fetched 32 B in 30s (1 B/s)
Reading package lists... Done
E: Failed to fetch http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/InRelease Clearsigned file isn't valid, got 'NOSPLIT' (does the network require authentication?)
E: Some index files failed to download. They have been ignored, or old ones used instead.
解决办法:
修改/etc/apt/sources.list.d目录下
cuda.list
原始内容为:
deb http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64 /
修改为:
deb https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64 /
ps: http 修改为https
1.2上述问题解决后 执行sudo apt update 报以下错误
Get:2 http://cn.archive.ubuntu.com/ubuntu xenial-updates/main amd64 linux-image-4.15.0-122-generic amd64 4.15.0-122.124~16.04.1 [7996 kB]
Fetched 21.1 MB in 24s (860 kB/s)
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = "en_HK:en",
LC_ALL = (unset),
LANG = "en_HK.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to the standard locale ("C").
locale: Cannot set LC_ALL to default locale: No such file or directory
(Reading database ... 722980 files and directories currently installed.)
Preparing to unpack .../linux-modules-4.15.0-122-generic_4.15.0-122.124~16.04.1_amd64.deb ...
Unpacking linux-modules-4.15.0-122-generic (4.15.0-122.124~16.04.1) ...
dpkg: error processing archive /var/cache/apt/archives/linux-modules-4.15.0-122-generic_4.15.0-122.124~16.04.1_amd64.deb (--unpack):
cannot copy extracted data for './boot/config-4.15.0-122-generic' to '/boot/config-4.15.0-122-generic.dpkg-new': failed to write (No space left on device)
No apport report written because the error message indicates a disk full error
dpkg-deb: error: subprocess paste was killed by signal (Broken pipe)
Preparing to unpack .../linux-image-4.15.0-122-generic_4.15.0-122.124~16.04.1_amd64.deb ...
Unpacking linux-image-4.15.0-122-generic (4.15.0-122.124~16.04.1) ...
dpkg: error processing archive /var/cache/apt/archives/linux-image-4.15.0-122-generic_4.15.0-122.124~16.04.1_amd64.deb (--unpack):
cannot copy extracted data for './boot/vmlinuz-4.15.0-122-generic' to '/boot/vmlinuz-4.15.0-122-generic.dpkg-new': failed to write (No space left on device)
No apport report written because the error message indicates a disk full error
dpkg-deb: error: subprocess paste was killed by signal (Broken pipe)
Errors were encountered while processing:
/var/cache/apt/archives/linux-modules-4.15.0-122-generic_4.15.0-122.124~16.04.1_amd64.deb
/var/cache/apt/archives/linux-image-4.15.0-122-generic_4.15.0-122.124~16.04.1_amd64.deb
E: Sub-process /usr/bin/dpkg returned an error code (1)
参考https://blog.csdn.net/jinguangliu/article/details/78145423
执行修复命令,并重新执行更新和升级,确保完整修复问题。
sudo apt --fix-broken install
sudo apt-get update
sudo apt-get upgrade
执行第一条指令时又报新错误
Get:2 http://cn.archive.ubuntu.com/ubuntu xenial-updates/main amd64 linux-image-4.15.0-122-generic amd64 4.15.0-122.124~16.04.1 [7996 kB]
Fetched 21.1 MB in 24s (860 kB/s)
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = "en_HK:en",
LC_ALL = (unset),
LANG = "en_HK.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to the standard locale ("C").
locale: Cannot set LC_ALL to default locale: No such file or directory
(Reading database ... 722980 files and directories currently installed.)
Preparing to unpack .../linux-modules-4.15.0-122-generic_4.15.0-122.124~16.04.1_amd64.deb ...
Unpacking linux-modules-4.15.0-122-generic (4.15.0-122.124~16.04.1) ...
dpkg: error processing archive /var/cache/apt/archives/linux-modules-4.15.0-122-generic_4.15.0-122.124~16.04.1_amd64.deb (--unpack):
cannot copy extracted data for './boot/config-4.15.0-122-generic' to '/boot/config-4.15.0-122-generic.dpkg-new': failed to write (No space left on device)
No apport report written because the error message indicates a disk full error
dpkg-deb: error: subprocess paste was killed by signal (Broken pipe)
Preparing to unpack .../linux-image-4.15.0-122-generic_4.15.0-122.124~16.04.1_amd64.deb ...
Unpacking linux-image-4.15.0-122-generic (4.15.0-122.124~16.04.1) ...
dpkg: error processing archive /var/cache/apt/archives/linux-image-4.15.0-122-generic_4.15.0-122.124~16.04.1_amd64.deb (--unpack):
cannot copy extracted data for './boot/vmlinuz-4.15.0-122-generic' to '/boot/vmlinuz-4.15.0-122-generic.dpkg-new': failed to write (No space left on device)
No apport report written because the error message indicates a disk full error
dpkg-deb: error: subprocess paste was killed by signal (Broken pipe)
Errors were encountered while processing:
/var/cache/apt/archives/linux-modules-4.15.0-122-generic_4.15.0-122.124~16.04.1_amd64.deb
/var/cache/apt/archives/linux-image-4.15.0-122-generic_4.15.0-122.124~16.04.1_amd64.deb
E: Sub-process /usr/bin/dpkg returned an error code (1)
显示failed to write (No space left on device)
df -h 查看 是/dev/sda1 磁盘满了 导致的问题,可以清理以下对应空间,之后问题就解决了
ps:把/boot目录清理一下,手动删除了/boot目录下一些非*4.4.0-178*的文件
2.安装openmpi
Open MPI:
到官网下载openmpi-x.x.x.tar.gz
tar -xvzf openmpi-x.x.x.tar.gz
cd openmpi-x.x.x
./configure --prefix="/usr/local/openmpi"
make -j 8
sudo make install
在.bashrc中添加环境变量
export PATH="$PATH:/usr/local/cuda/bin:/usr/local/openmpi/bin"
export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:/usr/local/cuda/lib64:/usr/local/openmpi/lib/"
source ~/.bashrc
打开新的终端使之生效。
3.安装horodvod
源码编译或者直接pip安装:一定要好好看官网教程,一开始也是看各种网上教程,但是都没有成功,最后还是跟着官方教程安装成功。
不过其中也有一个比较重要的原因就是horovod对cuda mxnet,tf,pytorch各种版本也有一定要求,要注意
此次是在ubuntu18.04 x64-cuda10.0 -pytorch1.6.0-mxnet1.5.0下安装成功
需要注意的是,如果服务器上有多个版本的cuda,并且你所使用的并非软链接到/usr/local/cuda的版本,那么需要设置cuda路径等变量:
$ HOROVOD_GPU_ALLREDUCE=NCCL HOROVOD_CUDA_HOME=/usr/local/cuda-9.0 HOROVOD_NCCL_HOME=xxx-nccl-path pip install --no-cache-dir horovod
否则可能出现计算能力不匹配等问题。