GPU加入k8s集群时，初始化显卡容器错误“**Error error validating toolkit installation: exec: \“nvidia-smi\“:executable

问题现象：

GPU加入k8s集群时，初始化显卡容器错误"Error error validating toolkit installation: exec: “nvidia-smi”:executable file not found in $PATH toolkiy is not ready

查看pod状态：
kubectl get pods -n gpu-operator-resources

NAME READY STATUS RESTARTS AGE
nvidia-dcgm-x5rqs 0/1 Init:0/1 2 6d20h
nvidia-device-plugin-daemonset-jhjhb 0/1 Init:0/1 0 6d20h
gpu-feature-discovery-pd4xv 0/1 Init:0/1 2 6d20h
nvidia-dcgm-exporter-7mjgt 0/1 Init:0/1 2 6d20h
nvidia-operator-validator-9xjmv 0/1 Init:Error 10 26m

kubectl logs -n gpu-operator-resources -f nvidia-operator-validator-9xjmv -c toolkit-validation
time=“2021-11-18T09:29:24Z” level=info msg=“Error: error validating toolkit installation: exec: “nvidia-smi”: executable file not found in $PATH” toolkit is not ready

基本环境相关：

Ubuntu 20.04
Kubernetes 1.21.6
CUDA 11.4

解决方法：

按照nvidia-docker 安装方法，设置docker的默认运行时，
在daemon.json文件中添加如下内容
vim /etc/docker/daemon.json
{
“default-runtime”: “nvidia”
“runtimes”: {
“nvidia”: {
“path”: “/usr/bin/nvidia-container-runtime”,
“runtimeArgs”: []
}
}
}
#重启docker服务即可生效
systemctl restart docker

等待一段时间，查看pod状态是否恢复。