Linux 分布式搭建
第一步:在Linux上安装虚拟环境virtualenv
1、安装virtualenv
pip3 install virtualenv
2、安装 virtualenv的扩展管理包virtualwrapper,用于更方便管理虚拟环境
pip3 install virtualenvwrapper
3、配置环境变量
vim ~/.bashrc
创建目录用来存放虚拟环境
mkdir $HOME/.virtualenvs
打开根目录下的.bashrc文件,在文件中添加以下三行代码
export WORKON_HOME=$HOME/.virtualenvs # 指定虚拟环境的工作目录
export VIRTUALENVWRAPPER_PYTHON=/usr/local/python3/bin/python3.6 # 指定使用的python版本
source /usr/local/python3/bin/virtualenvwrapper.sh # 指定脚本virtualenvwrapper.sh的位置
以上文件目录视具体情况而定,virtualenvwrapper.sh一般在你当前版本的python安装目录下,修改保存.bashrc文件后,使用命令
source ~/.bashrc
使之生效,生成必要的文件。
4、接着就可以开始使用虚拟环境,以下是使用virtalenv的一些常用命令:
mkvirtualenv [虚拟环境名称] #创建虚拟环境
workon [虚拟环境名称] # 激活虚拟环境
deactivate # 退出虚拟环境
lsvirtualenv # 列出全部的虚拟环境
rmvirtualenv [虚拟机环境名称] # 删除指定虚拟环境
cdvirtualenv # 进入虚拟环境目录
lssitepackages # 列出虚拟环境的包
第二步:安装NCCL2 https://docs.nvidia.com/deeplearning/sdk/nccl-install-guide/index.html
1、下载系统对应版本的NCCL2 https://developer.nvidia.com/nccl , 分为local版本和network版本
2、以Centos版本为例:
-
Install the repository.
-
For the local NCCL repository:
sudo rpm -i nccl-repo-<version>.rpm
-
For the network repository:
sudo rpm -i nvidia-machine-learning-repo-<version>.rpm
-
-
Update the YUM database:
sudo yum update
-
Install the libnccl2 package with YUM. Additionally, if you need to compile applications with NCCL, you can install the libnccl-devel package and optionally the libnccl-static package if you intend to link NCCL statically in your application:
Note: If you are using the network repository, the following command will upgrade CUDA to the latest version.
sudo yum install libnccl libnccl-devel libnccl-static
If you prefer to keep an older version of CUDA, specify a specific version, for example:
sudo yum install libnccl-2.4.8-1+cuda10.0 libnccl-devel-2.4.8-1+cuda10.0 libnccl-static-2.4.8-1+cuda10.0
第三步:安装openmpi
1、下载某一个版本的openmpi https://www.open-mpi.org/software/ompi/v4.0/
2、解压并按照以下步骤安装
shell$ gunzip -c openmpi-4.0.2.tar.gz | tar xf -
shell$ cd openmpi-4.0.2
shell$ ./configure --prefix=/usr/local
<...lots of output...>
shell$ make all install
第四步:安装tensorflow-gpu
pip3 install tensorflow-gpu==1.12
第五步:安装horovod
Install the horovod
pip package.
If you installed NCCL 2 using the nccl-.txz
package, you should specify the path to NCCL 2 using the HOROVOD_NCCL_HOME
environment variable.
$ HOROVOD_NCCL_HOME=/usr/local/nccl-<version> HOROVOD_GPU_ALLREDUCE=NCCL pip install --no-cache-dir horovod
If you installed NCCL 2 using the Ubuntu package, you can run:
$ HOROVOD_GPU_ALLREDUCE=NCCL pip install --no-cache-dir horovod
If you installed NCCL 2 using the CentOS / RHEL package, you can run:
$ HOROVOD_NCCL_INCLUDE=/usr/include HOROVOD_NCCL_LIB=/usr/lib64 HOROVOD_GPU_ALLREDUCE=NCCL pip install --no-cac
【注】pytorch安装之后通常汇报 not import name ssl的问题,解决方案是重新编译python