问题现象:
GPU加入k8s集群时,初始化显卡容器错误"Error error validating toolkit installation: exec: “nvidia-smi”:executable file not found in $PATH toolkiy is not ready
查看pod状态:
kubectl get pods -n gpu-operator-resources
NAME READY STATUS RESTARTS AGE
nvidia-dcgm-x5rqs 0/1 Init:0/1 2 6d20h
nvidia-device-plugin-daemonset-jhjhb 0/1 Init:0/1 0 6d20h
gpu-feature-discovery-pd4xv 0/1 Init:0/1 2 6d20h
nvidia-dcgm-exporter-7mjgt 0/1 Init:0/1 2 6d20h
nvidia-operator-validator-9xjmv 0/1 Init:Error 10 26m
kubectl logs -n gpu-operator-resources -f nvidia-operator-validator-9xjmv -c toolkit-validation
time=“2021-11-18T09:29:24Z” level=info msg=“Error: error validating toolkit installation: exec: “nvidia-smi”: executable file not found in $PATH” toolkit is not ready
基本环境相关:
Ubuntu 20.04
Kubernetes 1.21.6
CUDA 11.4
解决方法:
按照nvidia-docker 安装方法,设置docker的默认运行时,
在daemon.json文件中添加如下内容
vim /etc/docker/daemon.json
{
“default-runtime”: “nvidia”
“runtimes”: {
“nvidia”: {
“path”: “/usr/bin/nvidia-container-runtime”,
“runtimeArgs”: []
}
}
}
#重启docker服务即可生效
systemctl restart docker
等待一段时间,查看pod状态是否恢复。