监控不携带metrics接口的服务
第一部分：监控携带metrics的服务
普罗米修斯监控携带metric接口的服务的流程
1、通过EndPrints获取需要监控的ETCD的地址
2、创建Service，给予prometheus集群内部的ServiceMoniter使用
3、创建ServiceMoniter服务，进行服务注入，同时，部署需要访问证书
4、重启普罗米修斯监控Pod,载入监控项(实质更新prometheus-k8s-0)
5、加入grafnan做大屏显示
第二部分：
1.部署流程
1、部署expertor，从而创建一个metrics接口
2、部署EndPrints，链接expertor暴露出来的metrics接口
3、部署Service，基于ServiceMonitor使用
4、创建ServiceMonitor，注入promethues
5、测试
6、加入grafana，做大屏展示
2.部署一个测试环境
###安装nginx和yum仓库
[root@jenkins01 ~]# vim /etc/nginx/conf.d/default.conf ##立面加入下面的内容，再访问http://192.168.15.61/status成功出现了nginx的基本状态信息
location = /status {
stub_status;
}
3.部署expertor，从而创建一个metrics接口
3.1 部署expertor
###创建一个服务，获取并格式化metrics接口数据
[root@k8s-master-01 ~]# mkdir nginx-exporter
[root@k8s-master-01 ~]# cd nginx-exporter/
[root@k8s-master-01 nginx-exporter]# ll
总用量 0
[root@k8s-master-01 nginx-exporter]# vim exporter-nginx-deployment.yaml
kind: Deployment
apiVersion: apps/v1
metadata:
name: nginx-prometheus-exporter
spec:
selector:
matchLabels:
app: nginx-prometheus-exporter
deploy: nginx-exporter
template:
metadata:
labels:
app: nginx-prometheus-exporter
deploy: nginx-exporter
spec:
containers:
- name: nginx-exporter
image: nginx/nginx-prometheus-exporter:0.9.0
imagePullPolicy: IfNotPresent
command:
- "nginx-prometheus-exporter"
- "-nginx.scrape-uri=http://192.168.15.61/status"
- "/bin/bash"
- "-ce"
- "tail -f /dev/null"
~
##检测创建exporter成功
[root@k8s-master-01 nginx-exporter]# kubectl get pods
NAME READY STATUS RESTARTS AGE
nginx-prometheus-exporter-c5f4c69c-g5phb 1/1 Running 0 11s
4. 创建一个Service，给予prometheus集群获取格式化好了的metrics接口服务
[root@k8s-master-01 nginx-exporter]# vi exporter-nginx-svc.yaml
kind: Service
apiVersion: v1
metadata:
name: nginx-prometheus-exporter
spec:
ports:
- port: 9113 ###nginx导出器exporter的默认抓取端口就是9113
targetPort: 9113
name: nginx-prometheus-exporter
protocol: TCP
selector:
app: nginx-prometheus-exporter
deploy: nginx-exporter
##部署svc
[root@k8s-master-01 nginx-exporter]# kubectl apply -f exporter-nginx-svc.yaml
service/nginx-prometheus-exporter created
[root@k8s-master-01 nginx-exporter]# kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
nginx-prometheus-exporter ClusterIP 10.108.98.92 <none> 9113/TCP 6s
##测试SVC是否可用，结果可用
[root@k8s-master-01 nginx-exporter]# curl 10.108.98.92:9113/metrics
# HELP nginx_connections_writing Connections where NGINX is writing the response back to the client
# TYPE nginx_connections_writing gauge
nginx_connections_writing 1
# HELP nginx_http_requests_total Total http requests
# TYPE nginx_http_requests_total counter
nginx_http_requests_total 22500
nginx_up 1
# HELP nginxexporter_build_info Exporter build information
# TYPE nginxexporter_build_info gauge
nginxexporter_build_info{commit="5f88afbd906baae02edfbab4f5715e06d88538a0",date="2021-03-22T20:16:09Z",version="0.9.0"} 1
....出现一堆东西。。。。。
5.部署EndPoints，链接expertor暴露出来的metrics接口
# 参考上述Service，此处无需重复创建。
[root@k8s-master-01 ~]# kubectl get endpoints -A
NAMESPACE NAME ENDPOINTS AGE
18d
default nginx-prometheus-exporter 10.244.2.128:9113
[root@k8s-master-01 ~]# kubectl describe endpoints nginx-prometheus-exporter -n default
Name: nginx-prometheus-exporter
Namespace: default
Labels: <none>
Annotations: endpoints.kubernetes.io/last-change-trigger-time: 2021-04-27T16:05:31Z
Subsets:
Addresses: 10.244.2.128
NotReadyAddresses: <none>
Ports:
Name Port Protocol
---- ---- --------
nginx-prometheus-exporter 9113 TCP
Events: <none>
[root@k8s-master-01 ~]# kubectl get pod -n default
NAME READY STATUS RESTARTS AGE
nginx-prometheus-exporter-c5f4c69c-g5phb 1/1 Running 1 34h
6.部署Service，基于ServiceMonitor使用
# 参考上述Service，此处无需重复创建。
7.创建ServiceMonitor，注入prometheus
kind: ServiceMonitor
apiVersion: monitoring.coreos.com/v1
metadata:
labels:
app: nginx-prometheus-exporter-servicemonitor
deploy: nginx-exporter-servicemonitor
name: nginx-monitor
namespace: monitoring
spec:
endpoints:
- interval: 3s
port: nginx-prometheus-exporter
selector:
matchLabels:
app: nginx-prometheus-exporter
deploy: nginx-exporter
namespaceSelector:
matchNames:
- "default"
##部署servicemonitor
[root@k8s-master-01 nginx-exporter]# kubectl apply -f exporter-nginx-servicemonitor.yaml
servicemonitor.monitoring.coreos.com/nginx-monitor created
##检测servicemonitor，部署成功
[root@k8s-master-01 nginx-exporter]# kubectl get servicemonitor -n monitoring
NAME AGE
nginx-monitor 24s
8.重新激活prometheus（实质就是删了重新部署），目的就是为了加载刚才我们前面部署的东西
[root@k8s-master-01 nginx-exporter]# kubectl get prometheus -n monitoring
NAME VERSION REPLICAS AGE
k8s v2.15.2 2 2d16h
[root@k8s-master-01 nginx-exporter]# kubectl delete prometheus k8s -n monitoring
prometheus.monitoring.coreos.com "k8s" deleted
[root@k8s-master-01 nginx-exporter]# vi prometheus.yaml ##
kind: Prometheus
apiVersion: monitoring.coreos.com/v1
metadata:
labels:
prometheus: k8s
name: k8s
namespace: monitoring
spec:
alerting:
alertmanagers:
- name: alertmanager-main
namespace: monitoring
port: web
- name: alertmanager-main-etcd
namespace: kube-system
port: etcd
image: quay.io/prometheus/prometheus:v2.15.2
nodeSelector:
kubernetes.io/os: linux
podMonitorNamespaceSelector: {}
podMonitorSelector: {}
replicas: 2
resources:
requests:
memory: 400Mi
ruleSelector:
matchLabels:
prometheus: k8s
role: alert-rules
securityContext:
fsGroup: 2000
runAsNonRoot: true
runAsUser: 1000
serviceAccountName: prometheus-k8s
serviceMonitorNamespaceSelector: {}
serviceMonitorSelector: {}
version: v2.15.2
secrets:
- etcd-certs
[root@k8s-master-01 nginx-exporter]# kubectl apply -f prometheus.yaml
prometheus.monitoring.coreos.com/k8s created
9.检测prometheus安装成功与否
[root@k8s-master-01 nginx-exporter]# kubectl get pods -n monitoring
prometheus-k8s-0 3/3 Running 1 43s
prometheus-k8s-1 3/3 Running 1 43s
10.grafana展示
第三部分：监控Mysql
1、造环境
# 安装mariadb数据库,并授权
[root@k8s-master-01 opt]# yum install mariadb\* -y
[root@k8s-master-01 opt]# systemctl enable --now mariadb
[root@k8s-master-01 opt]# mysql
MariaDB [(none)]> grant select,replication client,process ON *.* to 'mysql_monitor'@'%' identified by 'oldboy@123';
Query OK, 0 rows affected (0.00 sec)
# (注意:授权ip为localhost，因为不是prometheus服务器来直接找mariadb 获取数据，而是prometheus服务器找mysql_exporter,mysql_exporter 再找mariadb。所以这个localhost是指的mysql_exporter的IP)
MariaDB [(none)]> flush privileges;
Query OK, 0 rows affected (0.01 sec)
MariaDB [(none)]> quit;
Bye
2、安装数据库exporter
---
kind: Deployment
apiVersion: apps/v1
metadata:
name: mysql-exporter-server
spec:
selector:
matchLabels:
k8s: mysql-exporter-server
template:
metadata:
labels:
k8s: mysql-exporter-server
spec:
containers:
- name: mysql
image: prom/mysqld-exporter
env:
- name: DATA_SOURCE_NAME
value: "mysql_monitor:oldboy@123@(mysql.default.svc.cluster.local:3306)"
ports:
- containerPort: 9104
---
kind: Service
apiVersion: v1
metadata:
name: mysql-exporter-server
labels:
k8s: mysql-exporter-server-svc
spec:
ports:
- port: 9104
targetPort: 9104
name: mysql-exporter-port
protocol: TCP
selector:
k8s: mysql-exporter-server
---
kind: ServiceMonitor
apiVersion: monitoring.coreos.com/v1
metadata:
labels:
app: mysql-prometheus-exporter-servicemonitor
deploy: mysql-exporter-servicemonitor
name: mysql-exporter-monitor
namespace: monitoring
spec:
endpoints:
- interval: 3s
port: mysql-exporter-port
selector:
matchLabels:
k8s: mysql-exporter-server-svc
namespaceSelector:
matchNames:
- "default"
第四部分：监控一个外部的exporter
1、造环境
[root@k8s-master-01 opt]# wget https://github.com/prometheus/mysqld_exporter/releases/download/v0.12.1/mysqld_exporter-0.12.1.linux-amd64.tar.gz
[root@k8s-master-01 opt]# tar -xf mysqld_exporter-0.12.1.linux-amd64.tar.gz
[root@k8s-master-01 opt]# mv mysqld_exporter-0.12.1.linux-amd64 /usr/local/mysqld_exporter-0.12.1
[root@k8s-master-01 opt]# ll /usr/local/mysqld_exporter-0.12.1
总用量 14484
-rw-r--r-- 1 3434 3434 11325 7月 29 2019 LICENSE
-rwxr-xr-x 1 3434 3434 14813452 7月 29 2019 mysqld_exporter
-rw-r--r-- 1 3434 3434 65 7月 29 2019 NOTICE
# 安装mariadb数据库,并授权
[root@k8s-master-01 opt]# yum install mariadb\* -y
[root@k8s-master-01 opt]# systemctl enable --now mariadb
[root@k8s-master-01 opt]# mysql
MariaDB [(none)]> grant select,replication client,process ON *.* to 'mysql_monitor'@'%' identified by 'oldboy@123';
Query OK, 0 rows affected (0.00 sec)
# (注意:授权ip为localhost，因为不是prometheus服务器来直接找mariadb 获取数据，而是prometheus服务器找mysql_exporter,mysql_exporter 再找mariadb。所以这个localhost是指的mysql_exporter的IP)
MariaDB [(none)]> flush privileges;
Query OK, 0 rows affected (0.01 sec)
MariaDB [(none)]> quit;
Bye
# 创建一个mariadb配置文件，写上连接的用户名与密码(和上面的授权的用户名 和密码要对应)
[root@k8s-master-01 opt]# vim /usr/local/mysqld_exporter-0.12.1/.my.cnf
[client]
user=mysql_monitor
password=oldboy@123
# 启动mysqld_exporter
[root@k8s-master-01 opt]# nohup /usr/local/mysqld_exporter-0.12.1/mysqld_exporter --config.my-cnf=/usr/local/mysqld_exporter-0.12.1/.my.cnf &
# 确认端口(9104)
2、创建EndPrints服务，绑定被监控的服务的IP
2、创建Service服务，提供服务给予ServiceMonitor使用
3、创建ServiceMonitor服务，进行服务注入
4、更新prometheus-k8s-0
5、测试
6、加入Grafana,做大屏展示。
扫一扫
2088
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
