MXNet结合kubeflow进行分布式训练

GPU集群配置MXNet+CUDA

为方便控制集群，写了脚本cmd2all.sh

#!/bin/bash
if [ $# -lt 3 ]; then
    echo "usage: $0 [type cmds hosts]"
    echo "for example: ./cmd2all.sh \"cmds\" \"touch t1.txt\" \"gpu1 gpu2\""
    echo "for example: ./cmd2all.sh \"path\" \"/home/gbxu/CUDA/" \"gpu1 gpu2\""
    exit -1;
fi
type=$1 # "cmds"
cmds_or_path=$2 # "touch test.txt"
#hosts=$3
hosts=(gpu10 gpu11 gpu12 gpu13 gpu14 gpu15 gpu16 gpu17 gpu18)
if [$type == "cmds"]
then
  for host in ${hosts[@]}
    do
      ssh $host nohup $cmds_or_path  &
  done
fi

if [$type == "path"]
then
  for host in ${hosts[@]}
    do
      nohup scp -r $cmds_or_path $host:~/ &
    done
fi

使用virtualenv

如果是python3的环境，需要virtualenv -p /usr/bin/python3 mxnetGPU
使用virtualenv，创建新的virtualenv，并修改.bashrc，使得在每次进入终端时activate虚拟环境（方便后期分布式运行）

hosts="gpu10 gpu11 gpu12 gpu13 gpu14 gpu15 gpu16 gpu17 gpu18 "
./cmd2all.sh "cmds" "sudo yum -y install epel-release && sudo yum -y install python-pip && sudo pip install virtualenv && virtualenv mxnetGPU" $hosts
./cmd2all.sh "cmds"  "echo \"## gbxu MXnet-GPU\" >> .bashrc" $hosts
./cmd2all.sh "cmds"  "echo \"source mxnetGPU/bin/activate\" >> .bashrc" $hosts

---

尝试在gpu10安装

Install NVIDIA Driver

本身已有驱动则该操作不必要。

lspci | grep -i nvidia #查看设备
modinfo nvidia #查看驱动
sudo yum -y remove nvidia-*
sudo sh  NVIDIA-Linux-x86_64-390.25.run #安装驱动

Install CUDA:

see documents:

• offline安装，online版本可能出现依赖缺失。
• 所有版本
  CUDA是NVIDIA推出的用于自家GPU的并行计算框架，只有当要解决的计算问题是可以大量并行计算的时候才能发挥CUDA的作用。
  下载： 见offline安装

#copy installer && run

# 若安装错误需要先卸载
sudo yum -y remove "cuda-*"
sudo rm -rf /usr/local/cuda*
sudo rpm -i cuda-repo-rhel7-9-2-local-9.2.148-1.x86_64.rpm
sudo yum clean all
sudo yum -y install cuda

gpu10利用yum local的安装出现问题，后来下载cuda_9.2.148_396.37_linux.run
sudo sh cuda_9.2.148_396.37_linux.run安装
并且在安装(or not, just try)时同意nvidia驱动，并且一路yes和default。
or, add /usr/local/cuda-9.2/lib64 to /etc/ld.so.conf and run ldconfig as root

添加CUDA环境变量

# export LD_LIBRARY_PATH=/usr/local/cuda/lib64/:$LD_LIBRARY_PATH
echo -e "export LD_LIBRARY_PATH=/usr/local/cuda/lib64/:\$LD_LIBRARY_PATH" >> .bashrc 
# export PATH=$PATH:/usr/local/cuda/bin
echo -e "export PATH=\$PATH:/usr/local/cuda/bin" >> .bashrc