GPU服务器docker启动失败问题解决

centos7 服务升级内核到4.4以后，执行docker run报错：

docker: Error response from daemon: OCI runtime create failed: unable to retrieve OCI runtime error (open /run/containerd/io.containerd.runtime.v1.linux/moby/fed8039bea12d7d6e31da38c3466459b8aab55c7fe191c82774ec11b2ea870a7/log.json: no such file or directory): exec: "nvidia-container-runtime": executable file not found in $PATH: unknown.
ERRO[0163] error waiting for container: context canceled 

执行nvidia-container-cli -k -d /dev/tty info命令检查，提示如下：

-- WARNING, the following logs are for debugging purposes only --

I0819 02:31:17.719753 137864 nvc.c:282] initializing library context (version=1.2.0, build=d22237acaea94aa5ad5de70aac903534ed598819)
I0819 02:31:17.720023 137864 nvc.c:256] using root /
I0819 02:31:17.720043 137864 nvc.c:257] using ldcache /etc/ld.so.cache
I0819 02:31:17.720061 137864 nvc.c:258] using unprivileged user 65534:65534
I0819 02:31:17.720349 137864 nvc.c:299] attempting to load dxcore to see if we are running under Windows Subsystem for Linux (WSL)
I0819 02:31:17.720549 137864 nvc.c:301] dxcore initialization failed, continuing assuming a non-WSL environment
I0819 02:31:17.736760 137874 nvc.c:192] loading kernel module nvidia
E0819 02:31:17.757157 137874 nvc.c:194] could not load kernel module nvidia
I0819 02:31:17.757210 137874 nvc.c:204] loading kernel module nvidia_uvm
E0819 02:31:17.775261 137874 nvc.c:206] could not load kernel module nvidia_uvm
I0819 02:31:17.775305 137874 nvc.c:212] loading kernel module nvidia_modeset
E0819 02:31:17.792443 137874 nvc.c:214] could not load kernel module nvidia_modeset
I0819 02:31:17.793337 137887 driver.c:101] starting driver service
I0819 02:31:17.831025 137864 driver.c:196] driver service terminated with signal 15
nvidia-container-cli: initialization error: nvml error: driver not loaded

说明GPU的驱动没有正确安装，应该是内核升级后，原有的驱动失效了。

执行uname -a输出结果为：

4.4.232-1.el7.elrepo.x86_64 #1 SMP Fri Jul 31 11:49:26 EDT 2020 x86_64 x86_64 x86_64 GNU/Linux

解决方法是卸载掉原有的kernel-header，安装新版本的header和devel包：

yum remove kernel-header -y
wget http://ftp.osuosl.org/pub/elrepo/kernel/el7/x86_64/RPMS/kernel-lt-devel-4.4.232-1.el7.elrepo.x86_64.rpm
wget http://ftp.osuosl.org/pub/elrepo/kernel/el7/x86_64/RPMS/kernel-lt-headers-4.4.232-1.el7.elrepo.x86_64.rpm
rpm -ivh kernel-lt-headers-4.4.232-1.el7.elrepo.x86_64.rpm  kernel-lt-devel-4.4.232-1.el7.elrepo.x86_64.rpm

之后，重新安装nvidia的驱动和cuda：

wget https://developer.download.nvidia.cn/compute/cuda/repos/rhel7/x86_64/cuda-repo-rhel7-10.1.243-1.x86_64.rpm
wget https://dl.fedoraproject.org/pub/epel/7/x86_64/Packages/e/epel-release-7-12.noarch.rpm
rpm -ivh cuda-repo-rhel7-10.1.243-1.x86_64.rpm
rpm -ivh epel-release-7-12.noarch.rpm
yum clean all
yum -y install nvidia-driver-latest-dkms cuda cuda-drivers 

最后执行：

docker run --gpus '"device=1,2"' nvidia/cuda:10.0-base nvidia-smi

显示正常