k8s学习笔记6-搭建监控kube-prometheus

一.介绍

应用部署到k8s上去了之后，我们想监控应用的pod，service以及节点的情况，目前k8s上提供的方案是kube-prometheus，这个方案整合了prometheus监控，报警，grafana仪表显示，感觉很好用

二.部署

1.源文件下载

#文件下载
git clone https://github.com/prometheus-operator/kube-prometheus.git
#进入到源文件夹,并且将相关部署文件拷贝到main文件夹下
cd kube-prometheus/manifests/
mkdir main
mv *.* main

2.镜像文件下载

1.由于网络原因，很多镜像无法下载，先查看当前有那些镜像需要下载

root@k8s-master1:~/kube-prometheus/manifests/main# grep image: ./*.*
./alertmanager-alertmanager.yaml:  image: quay.io/prometheus/alertmanager:v0.24.0
./blackboxExporter-deployment.yaml:        image: quay.io/prometheus/blackbox-exporter:v0.21.1
./blackboxExporter-deployment.yaml:        image: jimmidyson/configmap-reload:v0.5.0
./blackboxExporter-deployment.yaml:        image: quay.io/brancz/kube-rbac-proxy:v0.13.0
./grafana-deployment.yaml:        image: grafana/grafana:9.0.4
./kubeStateMetrics-deployment.yaml:        image: k8s.gcr.io/kube-state-metrics/kube-state-metrics:v2.5.0
./kubeStateMetrics-deployment.yaml:        image: quay.io/brancz/kube-rbac-proxy:v0.13.0
./kubeStateMetrics-deployment.yaml:        image: quay.io/brancz/kube-rbac-proxy:v0.13.0
./nodeExporter-daemonset.yaml:        image: quay.io/prometheus/node-exporter:v1.3.1
./nodeExporter-daemonset.yaml:        image: quay.io/brancz/kube-rbac-proxy:v0.13.0
./prometheus-prometheus.yaml:  image: quay.io/prometheus/prometheus:v2.37.0
./prometheusAdapter-deployment.yaml:        image: k8s.gcr.io/prometheus-adapter/prometheus-adapter:v0.9.1
./prometheusOperator-deployment.yaml:        image: quay.io/prometheus-operator/prometheus-operator:v0.58.0
./prometheusOperator-deployment.yaml:        image: quay.io/brancz/kube-rbac-proxy:v0.13.0
root@k8s-master1:~/kube-prometheus/manifests/main# 

2.在docker hub上将这些镜像下载，tag，推送到harbor，修改文件（只举一个列子）

#下载镜像
docker pull quay.io/prometheus/alertmanager:v0.24.0
#tag镜像
docker tag quay.io/prometheus/alertmanager:v0.24.0 registry.harbor.com/library/alertmanager:v0.24.0
#将镜像push到已经搭建的harbor仓库
docker push registry.harbor.com/library/alertmanager:v0.24.0
#修改对应的部署文件，将镜像地址替换
sed -i s#quay.io/prometheus/alertmanager:v0.24.0#registry.harbor.com/library/alertmanager:v0.24.0#g alertmanager-alertmanager.yaml 

3.部署

1.先部署setup里面的文件

root@k8s-master1:~/kube-prometheus/manifests/setup# kubectl create -f .

2.修改grafana-service.yaml文件，方便我们后续登录到grafana浏览器

piVersion: v1
kind: Service
metadata:
  labels:
    app.kubernetes.io/component: grafana
    app.kubernetes.io/name: grafana
    app.kubernetes.io/part-of: kube-prometheus
    app.kubernetes.io/version: 9.0.4
  name: grafana
  namespace: monitoring
spec:
  type: NodePort #修改成nodeport模式
  ports:
  - name: http
    port: 3000
    targetPort: http
    nodePort: 30180 #将端口设置成30180
  selector:
    app.kubernetes.io/component: grafana
    app.kubernetes.io/name: grafana
    app.kubernetes.io/part-of: kube-prometheus

3.部署main文件夹下的文件

root@k8s-master1:~/kube-prometheus/manifests/main# kubectl apply -f .

特别说明：
替换镜像的时候，对应的参数，也需要做对应的更改，不然会出现问题

root@k8s-master1:/root/kube-prometheus/manifests/main# grep v0.58.0 *.*
prometheusOperator-deployment.yaml:#        - --prometheus-config-reloader=quay.io/prometheus-operator/prometheus-config-reloader:v0.58.0
prometheusOperator-deployment.yaml:        - --prometheus-config-reloader=registry.harbor.com/library/prometheus-config-reloader:v0.58.0
prometheusOperator-deployment.yaml:        image: registry.harbor.com/library/prometheus-operator:v0.58.0
root@k8s-master1:/root/kube-prometheus/manifests/main# 

三.验证

验证是否部署成功

root@k8s-master1:~/kube-prometheus/manifests/main# kubectl get all -n monitoring 
NAME                                       READY   STATUS    RESTARTS   AGE
pod/alertmanager-main-0                    2/2     Running   0          4h21m
pod/alertmanager-main-1                    2/2     Running   0          4h21m
pod/alertmanager-main-2                    2/2     Running   0          4h21m
pod/blackbox-exporter-5fb779998c-8hgnz     3/3     Running   0          4h21m
pod/grafana-56f59b877b-7rhd7               1/1     Running   0          4h21m
pod/kube-state-metrics-c8bffc659-bjnw9     3/3     Running   0          4h21m
pod/node-exporter-55ghq                    2/2     Running   0          4h21m
pod/node-exporter-mbg6s                    2/2     Running   0          4h21m
pod/node-exporter-nnvqw                    2/2     Running   0          4h21m
pod/prometheus-adapter-68f86c4dc6-2zpkk    1/1     Running   0          4h21m
pod/prometheus-adapter-68f86c4dc6-tj56z    1/1     Running   0          4h21m
pod/prometheus-k8s-0                       2/2     Running   0          4h21m
pod/prometheus-k8s-1                       2/2     Running   0          4h21m
pod/prometheus-operator-58974d75dd-24g42   2/2     Running   0          4h21m

NAME                            TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                         AGE
service/alertmanager-main       ClusterIP   10.105.163.36   <none>        9093/TCP,8080/TCP               4h21m
service/alertmanager-operated   ClusterIP   None            <none>        9093/TCP,9094/TCP,9094/UDP      4h21m
service/blackbox-exporter       ClusterIP   10.106.6.24     <none>        9115/TCP,19115/TCP              4h21m
service/grafana                 NodePort    10.108.36.154   <none>        3000:30180/TCP                  4h21m
service/kube-state-metrics      ClusterIP   None            <none>        8443/TCP,9443/TCP               4h21m
service/node-exporter           ClusterIP   None            <none>        9100/TCP                        4h21m
service/prometheus-adapter      ClusterIP   10.105.230.63   <none>        443/TCP                         4h21m
service/prometheus-k8s          NodePort    10.109.111.46   <none>        9090:30181/TCP,8080:32154/TCP   4h21m
service/prometheus-operated     ClusterIP   None            <none>        9090/TCP                        4h21m
service/prometheus-operator     ClusterIP   None            <none>        8443/TCP                        4h21m

NAME                           DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR            AGE
daemonset.apps/node-exporter   3         3         3       3            3           kubernetes.io/os=linux   4h21m

NAME                                  READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/blackbox-exporter     1/1     1            1           4h21m
deployment.apps/grafana               1/1     1            1           4h21m
deployment.apps/kube-state-metrics    1/1     1            1           4h21m
deployment.apps/prometheus-adapter    2/2     2            2           4h21m
deployment.apps/prometheus-operator   1/1     1            1           4h21m

NAME                                             DESIRED   CURRENT   READY   AGE
replicaset.apps/blackbox-exporter-5fb779998c     1         1         1       4h21m
replicaset.apps/grafana-56f59b877b               1         1         1       4h21m
replicaset.apps/kube-state-metrics-c8bffc659     1         1         1       4h21m
replicaset.apps/prometheus-adapter-68f86c4dc6    2         2         2       4h21m
replicaset.apps/prometheus-operator-58974d75dd   1         1         1       4h21m

NAME                                 READY   AGE
statefulset.apps/alertmanager-main   3/3     4h21m
statefulset.apps/prometheus-k8s      2/2     4h21m
root@k8s-master1:~/kube-prometheus/manifests/main# 

四.登录grafana，并导入模板

1.登录grafana，默认帐号是admin，默认密码也是admin，登录后需要修改密码

1.主机详情的dashboard

a.导入主机详情的dashboard-1

b.导入主机详情的dashboard-2，点击import

c.进入到主机的dashboard界面，上面有很多地方，可以根据自己的需求去设置
界面1：

界面2

上面两个图，仅仅是主机k8s-master1主机的详细监控数据，可以根据自己的需求进行切换切换

2.pod,service,namespace,节点的dashboard

a.导入数据数据模板13105

b.进入到导入界面，选择import进行导入，也可以改名啥的

c.点击导入后，就会进入到dashboard界面
界面1：

目前界面暂时没有数据，可以暂时看一下，我们的测试集群



如果根据需求，有需要更换不同的dashboard，可以登录网址https://grafana.com/grafana/dashboards进行选择

五.参考链接

1.kubernetes通过kube-prometheus安装高可用的监控告警（官方中文文档）
https://www.orchome.com/8193
2.grafana的dashboard界面
https://grafana.com/grafana/dashboards
3.Prometheus监控（kube-prometheus）
https://blog.csdn.net/qq_16240085/article/details/117460715