Prometheus 基于pod服务发现监控tomcat

构建镜像

---

默认tomcat镜像不提供普罗米修斯相关的支持，所以需要自己制作镜像。 

在tomcat发布目录下需要有一个metrics.war，这样接口地址才能暴露tomcat的一些指标，以及采集指标的tomcat_exporter_client-0.0.12.jar

[root@master tomcat]# cat Dockerfile 
FROM tomcat
ADD metrics.war /usr/local/tomcat/webapps/
ADD simpleclient-0.8.0.jar  /usr/local/tomcat/lib/
ADD simpleclient_common-0.8.0.jar /usr/local/tomcat/lib/
ADD simpleclient_hotspot-0.8.0.jar /usr/local/tomcat/lib/
ADD simpleclient_servlet-0.8.0.jar /usr/local/tomcat/lib/
ADD tomcat_exporter_client-0.0.12.jar /usr/local/tomcat/lib/


[root@master tomcat]# docker build -t 'tomcat_prometheus:v1' .
Sending build context to Docker daemon    128kB
Step 1/7 : FROM tomcat
latest: Pulling from library/tomcat
5e0b432e8ba9: Pull complete 
a84cfd68b5ce: Pull complete 
e8b8f2315954: Pull complete 
Digest: sha256:429d5c140820d5eba38042823501d5f4e0f03b282133513e29c22d6358daa7c7
Status: Downloaded newer image for tomcat:latest
 ---> 24207ccc9cce
Step 2/7 : ADD metrics.war /usr/local/tomcat/webapps/
 ---> 65f67a4763ea
Step 3/7 : ADD simpleclient-0.8.0.jar  /usr/local/tomcat/lib/
 ---> 72d7414ff158
Step 4/7 : ADD simpleclient_common-0.8.0.jar /usr/local/tomcat/lib/
 ---> 8e58e81f047f
Step 5/7 : ADD simpleclient_hotspot-0.8.0.jar /usr/local/tomcat/lib/
 ---> 2324ee685ac8
Step 6/7 : ADD simpleclient_servlet-0.8.0.jar /usr/local/tomcat/lib/
 ---> deb785fb6740
Step 7/7 : ADD tomcat_exporter_client-0.0.12.jar /usr/local/tomcat/lib/
 ---> 92539e485933
Successfully built 92539e485933
Successfully tagged tomcat_prometheus:v1

放在公有仓库上面了。 

docker pull xianchao/tomcat_prometheus:v1 #在k8s的node节点拉取镜像拉取镜像

 

 

基于上面的镜像创建一个tomcat实例 

---

pod里面这个字段也就是说我们创建的pod可以被Prometheus抓取到。

      annotations:
        prometheus.io/scrape: 'true'

[root@master tomcat]# cat deploy.yaml 
apiVersion: apps/v1
kind: Deployment
metadata:
  name: tomcat-deployment
  namespace: monitor
spec:
  selector: 
    matchLabels: 
     app: tomcat
  replicas: 1 # tells deployment to run 2 pods matching the template
  template: # create pods using pod definition in this template
    metadata:
      labels:
        app: tomcat
      annotations:
        prometheus.io/scrape: 'true'
    spec:
      containers:
      - name: tomcat
        image: xianchao/tomcat_prometheus:v1
        ports:
        - containerPort: 8080
        securityContext: 
          privileged: true

[root@master prometheus]# kubectl get pod -n monitor -o wide
NAME                                 READY   STATUS    RESTARTS   AGE   IP              NODE     NOMINATED NODE   READINESS GATES
grafana-757fcd5f7c-n6sj7             1/1     Running   19         26d   10.233.96.236   node2    <none>           <none>
node-exporter-26849                  1/1     Running   21         36d   192.168.100.5   master   <none>           <none>
node-exporter-jbbcq                  1/1     Running   23         36d   192.168.100.6   node1    <none>           <none>
node-exporter-mh78s                  1/1     Running   21         36d   192.168.100.7   node2    <none>           <none>
prometheus-server-5775f99578-6pxsj   1/1     Running   0          89s   10.233.90.131   node1    <none>           <none>
tomcat-deployment-699d65b574-6zzcn   1/1     Running   0          10m   10.233.96.242   node2    <none>           <none>

我们可以看到 kubernetes-pods下面出现了很多实例，这是因为这里我们使用的是 pod 类型的服务发现，那么我们应该怎样来过滤出这个pod来呢？

keep，就是只把符合我们要求的给保留下来，哪些才是符合我们要求的呢？ 

我们可以把鼠标放置在任意一个 target 上，可以查看到Before relabeling里面所有的元数据 

    - job_name: 'kubernetes-pods-tomcat'
      kubernetes_sd_configs:
      - role: pod
      relabel_configs:
      - source_labels: [__meta_kubernetes_namespace,__meta_kubernetes_pod_label_app]
        action: keep
        regex: monitor;tomcat

可以看到最后被监控上了。 指标如下

[root@master prometheus]# curl http://10.233.96.242:8080/metrics/
# HELP jvm_classes_loaded The number of classes that are currently loaded in the JVM
# TYPE jvm_classes_loaded gauge
jvm_classes_loaded 2760.0
# HELP jvm_classes_loaded_total The total number of classes that have been loaded since the JVM has started execution
# TYPE jvm_classes_loaded_total counter
jvm_classes_loaded_total 2760.0
# HELP jvm_classes_unloaded_total The total number of classes that have been unloaded since the JVM has started execution
# TYPE jvm_classes_unloaded_total counter
jvm_classes_unloaded_total 0.0
# HELP process_cpu_seconds_total Total user and system CPU time spent in seconds.
# TYPE process_cpu_seconds_total counter
process_cpu_seconds_total 7.53
# HELP process_start_time_seconds Start time of the process since unix epoch in seconds.
# TYPE process_start_time_seconds gauge
process_start_time_seconds 1.639707011452E9

最后这个通过访问物理机ip和端口访问tomcat界面service可以创建也可以不创建。 

 cat tomcat-service.yaml
kind: Service  #service 类型
apiVersion: v1
metadata:
#  annotations:
#    prometheus.io/scrape: 'true'
  name: tomcat-service
spec:
  selector:
    app: tomcat
  ports:
  - nodePort: 31360
    port: 80
    protocol: TCP
    targetPort: 8080
  type: NodePort