参考文档:
https://github.com/NVIDIA/gpu-monitoring-tools
一、k8s环境
1、查看k8s集群
kubectl get nodes -o wide
2、查看k8s集群gpu所在节点
kubectl get nodes "-o=custom-columns=NAME:.metadata.name,GPU:.status.allocatable.nvidia\.com/gpu"
3、为有gpu的节点打标签 并查看
kubectl label nodes node0 hardware-type=NVIDIAGPU
kubectl label nodes node2 hardware-type=NVIDIAGPU
kubectl get nodes --show-labels
二、增加gpu监控
1、下载代码
git clone https://github.com/NVIDIA/gpu-monitoring-tools.git
2、打包镜像
cd gpu-monitoring-tools/exporters/prometheus-dcgm/k8s/pod-gpu-metrics-exporter/
sudo docker build -t pod-gpu-metrics-exporter .
cd gpu-monitoring-tools/exporters/prometheus-dcgm/dcgm-exporter/
sudo docker build -t dcgm-exporter .git
3、上传镜像到docker私有仓库
过程略
可参考https://blog.csdn.net/qq_29940863/article/details/89848075
4、修改启动文件镜像地址
cd gpu-monitoring-tools/exporters/prometheus-dcgm/k8s/pod-gpu-metrics-exporter/
vim pod-gpu-metrics-exporter-daemonset.yaml
改动见下图
5、启动
kubectl apply -f pod-gpu-metrics-exporter-daemonset.yaml
三、验证
1、查看集群
kubectl get ds -n monitoring
kubectl get po -n monitoring
注:monitoring(ns)和下图中没有红框的ds和po是在为k8s集群添加prometheus监控时所创建的
参考:https://github.com/coreos/kube-prometheus#quickstart
https://www.jianshu.com/p/2fbbe767870d
2、查看url
访问http://IP:9400/gpu/metrics
注:IP是node0和node2的地址
四、外部prometheus集成
1、编写配置文件
vim prometheus.yml
global:
scrape_interval: 60s
evaluation_interval: 60s
scrape_configs:
- job_name: prometheus
static_configs:
- targets: ['localhost:9090']
labels:
instance: prometheus
- job_name: prom_gpu
metrics_path: /gpu/metrics
static_configs:
- targets: ['IP:9400', 'IP:9400']
labels:
instance: gpu
2、启动镜像
sudo docker run --name prometheus -d -p 9090:9090 -v prometheus.yml:/etc/prometheus/prometheus.yml quay.io/prometheus/prometheus
3、访问
http://IP:9090
五、其它
1、节点环境
gpu所在节点可能需要安装docker-compose
参考:https://docs.docker.com/compose/install/
2、下载
https://download.csdn.net/download/qq_29940863/12588524