Prometheus监控案例-tomcat、mysql、redis、haproxy、nginx

监控tomcat

tomcat自身并不能提供监控指标数据，需要借助第三方exporter实现：https://github.com/nlighten/tomcat_exporter

构建镜像

基于tomcat官方镜像，重新制作一个镜像，将tomcat-exporter和tomcat整合到一起。Ddockerfile如下：

FROM tomcat:8.5.73
LABEL maintainer="admin@163.com"

ADD server.xml /usr/local/tomcat/conf/

RUN mkdir -p /data/tomcat/webapps


ADD myapp /data/tomcat/webapps/myapp

#添加tomcat-expoter相关文件
ADD metrics.war  /data/tomcat/webapps
ADD simpleclient-0.8.0.jar /usr/local/tomcat/lib/
ADD simpleclient_common-0.8.0.jar /usr/local/tomcat/lib/
ADD simpleclient_hotspot-0.8.0.jar /usr/local/tomcat/lib/
ADD simpleclient_servlet-0.8.0.jar /usr/local/tomcat/lib/
ADD tomcat_exporter_client-0.0.17.jar /usr/local/tomcat/lib/

EXPOSE 8080 8443

ADD run_tomcat.sh /usr/bin/

CMD ["/usr/bin/run_tomcat.sh"]

myapp/目录下是一个测试页面index.jsp

root@master-01:~/dockerfile/tomcat# cat myapp/index.jsp
tomcat myapp

tomcat配置文件server.xml中需要指定app路径，并且打开war包的自动解压和部署功能

root@master-01:~/dockerfile/tomcat# cat server.xml |grep /data/tomcat
      <Host name="localhost"  appBase="/data/tomcat/webapps"  unpackWARs="true" autoDeploy="true">

run_tomcat.sh是启动tomcat脚本

root@master-01:~/dockerfile/tomcat# cat run_tomcat.sh
#!/bin/bash
/usr/local/tomcat/bin/catalina.sh start

tail -f /etc/hosts

执行构建，上传到私有仓库

docker build -t harbor-server.linux.io/n70/tomcat-with-exporter:8.5.73 .
docker push harbor-server.linux.io/n70/tomcat-with-exporter:8.5.73

运行一个测试容器，验证metrics接口是否可以访问

docker run -d --name tomcat -p 8080:8080 harbor-server.linux.io/n70/tomcat-with-exporter:8.5.73

部署tomcat

以Deployment方式运行tomcat

apiVersion: apps/v1
kind: Deployment
metadata:
  name: tomcat-deployment
  namespace: default
spec:
  selector:
    matchLabels:
     app: tomcat
  replicas: 1
  template:
    metadata:
      labels:
        app: tomcat
      annotations:
        prometheus.io/scrape: 'true'
        prometheus.io/port: "8080"
    spec:
      containers:
      - name: tomcat
        image: harbor-server.linux.io/n70/tomcat-with-exporter:8.5.73
        imagePullPolicy: Always
        ports:
        - containerPort: 8080
        securityContext:
          privileged: true
---
kind: Service
apiVersion: v1
metadata:
  annotations:
    prometheus.io/scrape: 'true'
    prometheus.io/port: '8080'
  name: tomcat-service
spec:
  selector:
    app: tomcat
  ports:
  - nodePort: 31080
    port: 80
    protocol: TCP
    targetPort: 8080
  type: NodePort

将此yaml文件创建到集群上，然后等待Pod就绪

配置Prometheus采集数据

如果是部署在集群内部的Prometheus，建议使用角色为pod或endpoints的基于API Server的服务发现来发现target并监控。具体job配置可以参考之前的文章《Prometheus服务发现》

如果是集群外的Prometheus，添加一个job，内容如下：

- job_name: tomcat-exporter
    static_configs:
    - targets: ["192.168.122.12:31080"]

配置修改完成后，重启Prometheus。然后通过Prometheus界面查看target状态

在Grafana上导入模板，查看监控数据。模板地址：https://github.com/nlighten/tomcat_exporter/blob/master/dashboard/example.json

监控mysql

Prometheus官方提供了mysql-expoter用于采集mysql指标数据，项目地址：https://github.com/prometheus/mysqld_exporter

部署mysql

直接以apt方式安装mysql

apt -y install mariadb-server
mysql

mysql授权

创建一个账户并授权，之后mysql-exporter会使用此账户来访问mysql查询相关信息

> CREATE USER 'mysql-exporter'@'localhost' IDENTIFIED BY 'Admin@123' WITH MAX_USER_CONNECTIONS 3;
> GRANT PROCESS, REPLICATION CLIENT, SELECT ON *.* TO 'mysql-exporter'@'localhost';

mysql -u mysql-exporter -p -h localhost;	#验证权限

部署mysql-exporter

下载安装包

wget https://github.com/prometheus/mysqld_exporter/releases/download/v0.14.0/mysqld_exporter-0.14.0.linux-amd64.tar.gz
tar xf mysqld_exporter-0.14.0.linux-amd64.tar.gz
mv mysqld_exporter-0.14.0.linux-amd64/mysqld_exporter /usr/bin/
 mysqld_exporter -h #查看帮助信息

准备mysql-exporter配置文件

root@prometheus-node-02:~# mkdir /etc/mysqld-exporter
root@prometheus-node-02:~# cat /etc/mysqld-exporter/.my.cnf
[client]
#指定连接mysql的账户和密码
user = mysql-exporter
password = Admin@123

准备service文件

root@prometheus-node-02:~# cat /lib/systemd/system/mysql-exporter.service
[Unit]
Description=Prometheus Node Exporter
After=network.target

[Service]
ExecStart=/usr/bin/mysqld_exporter --config.my-cnf=/etc/mysql-exporter/.my.cnf --web.listen-address=192.168.122.23:9104

[Install]
WantedBy=multi-user.target

启动服务

systemctl daemon-reload
systemctl start mysql-exporter.service
systemctl status mysql-exporter.service
systemctl enable mysql-exporter.service

访问mysql-exporter指标接口测试

配置Prometheus采集数据

编辑Prometheus配置，添加一个job，内容如下：

  - job_name: mysql-exporter
    static_configs:
    - targets: ["192.168.122.23:9104"]

配置修改完成后，重启Prometheus。然后通过Prometheus界面查看target状态

Grafana导入模板，查看监控数据。模板地址：https://github.com/prometheus/mysqld_exporter/blob/main/mysqld-mixin/dashboards/mysql-overview.json

监控redis

redis-exporter项目地址：https://github.com/oliver006/redis_exporter

部署redis&redis-exporter

将redis和redis-exporter运行在同一个Pod中，然后以Deployment方式部署Pod

apiVersion: apps/v1
kind: Deployment
metadata:
  name: redis-deploy
spec:
  replicas: 1
  selector:
    matchLabels:
      app: redis
  template:
    metadata:
      labels:
        app: redis
      annotations:
        prometheus.io/scrape: "true"
        pometheus.io/port: "9121"
    spec:
      containers:
      - name: redis
        image: redis
        ports:
        - containerPort: 6379
        resources:
          requests:
            cpu: 100m
            memory: 100Mi
      - name: redis-exporter
        image: oliver006/redis_exporter
        ports:
        - containerPort: 9121
        resources:
          requests:
            cpu: 100m
            memory: 100Mi

kind: Service
apiVersion: v1
metadata:
  annotations:
    prometheus.io/scrape: 'true'
    prometheus.io/port: "9121"
  name: redis-exporter-service
spec:
  selector:
    app: redis
  ports:
  - nodePort: 31082
    name: prom
    port: 9121
    protocol: TCP
    targetPort: 9121
  type: NodePort

---
kind: Service
apiVersion: v1
metadata:
  name: redis-redis-service
spec:
  selector:
    app: redis
  ports:
  - nodePort: 31081
    name: redis
    port: 6379
    protocol: TCP
    targetPort: 6379
  type: NodePort

将yaml文件创建到集群上，等待Pod就绪

通过 任意节点:31082/metrics 访问redis-exporter指标接口测试

配置Prometheus采集数据

编辑Prometheus配置，添加job，内容如下：

  - job_name: redis-exporter
    static_configs:
    - targets: ["192.168.122.13:31082"]

配置修改完成后，重启Prometheus。然后通过Prometheus界面查看target状态

Grafana导入模板，查看监控数据。模板ID：14615

监控Haproxy

Prometheus官方提供了haproxy-exporter用于采集haproxy指标数据，项目地址：https://github.com/prometheus/haproxy_exporter

部署haproxy-exporter

下载安装包

wget https://github.com/prometheus/haproxy_exporter/releases/download/v0.15.0/haproxy_exporter-0.15.0.linux-amd64.tar.gz
tar xf haproxy_exporter-0.15.0.linux-amd64.tar.gz
mv haproxy_exporter-0.15.0.linux-amd64/haproxy_exporter /usr/bin/
haproxy_exporter -h		#查看帮助信息

haproxy-exporter可以通过两种方式连接haproxy来获取数据：

1. 通过socket文件连接haproxy获取数据，这种方式只能获取本机的haproxy数据
2. 通过haproxy状态页连接haproxy获取数据，这种方式可以获取本机或远程主机的数据，但需要在haproxy配置文件中开启状态页面配置

启动hapeoxy-exporter

#如果通过haproxy状态页获取数据，运行下面命令，其中haadmin:123456表示访问状态页的用户名密码
haproxy_exporter --haproxy.scrape-uri="http://haadmin:123456@127.0.0.1:9999/haproxy-status;csv" &

#如果通过socket文件获取数据，运行下面命令
haproxy_exporter --haproxy.scrape-uri="unix:/run/haproxy/admin.sock" &

访问haproxy-exporter指标接口验证，默认端口9101

配置Prometheus采集数据

编辑Prometheus配置添加job，内容如下：

  - job_name: haproxy-exporter
    static_configs:
      - targets: ["192.168.122.14:9101", "192.168.122.15:9101"]

配置修改完成后，重启Prometheus。然后通过Prometheus界面查看target状态

Grafana导入模板，查看监控数据。模板ID: 2428

监控nginx

nginx-exporter项目地址：https://github.com/hnlq715/nginx-vts-exporter

此nginx-exporter依赖一个nginx模块nginx-module-vts（https://github.com/vozlt/nginx-module-vts），所以需要编译安装nginx，并加载此模块。

nginx-module-vts模块会收集nginx指标数据，然后提供一个状态页面，nginx-exporter通过此状态页获取指标数据，再将数据转换为Prometheus需要的数据格式提供给Prometheus。

部署nginx

下载nginx源码包和nginx-module-vts模块

wget http://nginx.org/download/nginx-1.22.1.tar.gz
wget https://github.com/vozlt/nginx-module-vts/archive/refs/tags/v0.2.1.tar.gz
tar xf nginx-module-vts-0.2.1.tar.gz -C /usr/local/src/
tar xvf nginx-1.22.1.tar.gz
cd nginx-1.22.1/

./configure --prefix=/usr/local/nginx \
--with-http_ssl_module \
--with-http_v2_module \
--with-http_realip_module \
--with-http_stub_status_module \
--with-http_gzip_static_module \
--with-pcre \
--with-file-aio \
--with-stream \
--with-stream_ssl_module \
--with-stream_realip_module \
--add-module=/usr/local/src/nginx-module-vts-0.2.1/

修改nginx配置文件

http {
    vhost_traffic_status_zone;	#启用状态页

    ...

    server {

        ...

        location /status {	#状态页面location配置
            vhost_traffic_status_display;
            vhost_traffic_status_display_format html;
        }
    }
}

启动nginx

/usr/local/nginx/sbin/nginx -t
/usr/local/nginx/sbin/nginx

访问状态页测试

还可以用json格式输出数据

部署nginx-exporter

wget https://github.com/hnlq715/nginx-vts-exporter/releases/download/v0.10.3/nginx-vts-exporter-0.10.3.linux-amd64.tar.gz
tar xf nginx-vts-exporter-0.10.3.linux-amd64.tar.gz
cp nginx-vts-exporter-0.10.3.linux-amd64/nginx-vts-exporter /usr/bin/
nginx-vts-exporter  -nginx.scrape_uri http://192.168.122.23/status/format/json &	#启动nginx-exporter

访问nginx-exporter指标接口测试

配置Prometheus采集数据

编辑Prometheus配置，添加Job，内容如下：

  - job_name: nginx-exporter
    static_configs:
      - targets: ["192.168.122.23:9913"]

配置修改完成后，重启Prometheus。然后通过Prometheus界面查看target状态

Grafana导入模板，查看数据。模板地址：https://github.com/hnlq715/nginx-vts-exporter/blob/master/dashboard/nginx-vts-exporter.json

监控ingress-nginx-controller

ingress-nginx-controller程序自身可以提供指标数据，默认通过端口10254/metrics提供指标数据，在部署ingress-nginx-controller时候需要在容器上定义10254端口，并在ingress-nginx-controller对应的Service资源上也定义10254端口

        ports:
        - containerPort: 80
          name: http
          protocol: TCP
        - containerPort: 443
          name: https
          protocol: TCP
        - containerPort: 8443
          name: webhook
          protocol: TCP
        - name: metrics
          containerPort: 10254
          protocol: TCP

apiVersion: v1
kind: Service
metadata:
  labels:
    app.kubernetes.io/component: controller
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
    app.kubernetes.io/version: 1.3.1
  name: ingress-nginx-controller
  namespace: ingress-nginx
spec:
  ipFamilies:
  - IPv4
  ipFamilyPolicy: SingleStack
  ports:
  - appProtocol: http
    name: http
    port: 80
    protocol: TCP
    targetPort: http
  - appProtocol: https
    name: https
    port: 443
    protocol: TCP
    targetPort: https
  - name: metrics
    port: 10254
    targetPort: 10254
    protocol: TCP
  selector:
    app.kubernetes.io/component: controller
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/name: ingress-nginx
  type: NodePort

之后就可以通过nodePort访问ingress-nginx-controller的指标接口

然后在Prometheus中配置中添加Job，内容如下

  - job_name: nginx-ingress-controller
    static_configs:
      - targets: ["192.168.122.11:38144"]

配置修改完成后，重启Prometheus。通过Prometheus界面查看target状态

Grafana导入模板，查看监控数据。模板ID：9614