容器化应用安装jvm监控

容器化应用安装jvm监控

1.下载jvm监控jar包

wget https://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent/0.16.1/jmx_prometheus_javaagent-0.16.1.jar

2.编写jmx管理配置文件

cat prometheus-jmx-config.yaml
---

ssl: false 
lowercaseOutputName: false 
lowercaseOutputLabelNames: false 

3.编写dockerfile

#引用基础镜像
FROM harbor.xxx.xxx/tools/base-jdk:8-python-jdk-tini

#维护者信息
MAINTAINER xxx.xxx.xx
USER root

#环境变量设置
WORKDIR /opt/
ENV LANG="zh_CN.UTF-8"
ENV JAVA_OPTS=" -server -Xms2048m -Xmx4096m -Xmn768m -Xss512k -Djava.security.egd=file:/dev/./urandom "
ENV TZ=Asia/Shanghai LANG="C.UTF-8"
ENV JWT_AUTH_OPEN=true
ENV SW_AGENT_NAME=""
ENV SW_AGENT_COLLECTOR_BACKEND_SERVICES=""
ENV SW_AGENT_SPAN_LIMIT="200"
ENV SW_AGENT_OPTS=""
#添加JMX_OPTS
ENV JMX_OPTS="-javaagent:/opt/jmx/jmx_prometheus_javaagent-0.16.1.jar=8088:/opt/jmx/prometheus-jmx-config.yaml"
RUN mkdir /opt/jmx


#添加jar包
ADD ./*.jar /opt/ 
#agent.tar.gz是skywalking的agent代理包
ADD ./agent.tar.gz /opt/
#将jar包和配置文件打包进入容器
ADD ./jmx_prometheus_javaagent-0.16.1.jar /opt/jmx/
ADD ./prometheus-jmx-config.yaml /opt/jmx/

#容器启动时内部运行的命令
#启动命令添加$JMX_OPTS参数
ENTRYPOINT ["/sbin/tini", "--", "sh", "-c", "java $JAVA_OPTS $JMX_OPTS $SW_AGENT_OPTS -Dfile.encoding=utf-8 -jar /opt/*.jar"]

4.dockerignore添加屏蔽文件

!jmx_prometheus_javaagent-0.16.1.jar
!prometheus-jmx-config.yaml

5.gitlab对应项目devops上传jvm包和配置文件

jmx_prometheus_javaagent-0.16.1.jar
prometheus-jmx-config.yaml

6.cmdb 项目上线中添加环境变量及流水线

{“PLATFORM_TYPE”:“dev”,
“CONFIG_URL”:“nacos-headless.platform.svc.cluster.local:8848”,
“SW_AGENT_COLLECTOR_BACKEND_SERVICES”:“172.16.56.52:64386”,
“SW_AGENT_SPAN_LIMIT”:“200”,
“SW_AGENT_OPTS”:“-javaagent:./agent/skywalking-agent.jar”,
“JAVA_OPTS”:“’ -server -Xms2048m -Xmx8192m -Xmn768m -Xss512k -Djava.security.egd=file:/dev/./urandom -Dlog4j2.formatMsgNoLookups=true’”,
“JMX_OPTS”:“-javaagent:/opt/jmx/jmx_prometheus_javaagent-0.16.1.jar=8088:/opt/jmx/prometheus-jmx-config.yaml”
}

#jenkins流水线中
stage ('Build docker image and push to registry') 中添加

        sh 'cp devops/jmx_prometheus_javaagent-0.16.1.jar .'
        sh 'cp devops/prometheus-jmx-config.yaml .'

注意：sh 'cp target/*.jar .'拷贝到根目录，而docker构建的 ADD ./\*.jar /opt/ 拷贝到/opt下，这时如果jmx_prometheus_javaagent-0.16.1.jar也被拷贝到/opt下，那么启动时，可能优先启动jmx_prometheus_javaagent-0.16.1.jar这个jar而不是启动程序的jar，导致报错。那么dockerfile里面这个地方可以写成ADD ./trans\*.jar /opt/类似。

7.k8s prometheus添加监控job的形式

cat job-jmv.yaml
    - job_name: java
      scrape_interval: 5s
      kubernetes_sd_configs:
      - role: endpoints
        namespaces:
          names:
          - default
          - dev
          - test
      relabel_configs:
      - action: keep
        source_labels:
        - __meta_kubernetes_service_label_javaapp
        regex: java
      - action: keep
        source_labels:
        - __meta_kubernetes_endpoint_port_name
        regex: jmx-metrics

kubectl create cm jmx --from-file=job-jmv.yaml -nmonitoring

kubectl edit prometheus -nmonitoring prometheus-kube-prometheus-prometheus

  additionalScrapeConfigs:
    key: job-jmv.yaml
    name: jmx

8.修改chart配置，添加端口及标签。

对应项目中，需要修改service和deploy

deployment添加label标签：

javaapp: app

  labels:
    app: dev-bdp-jdbcexecutor
    chart: bdp-jdbcexecutor-sk-0.2.0
    heritage: Tiller
    javaapp: java

Service：

 apiVersion: v1
 kind: Service
 metadata:
   creationTimestamp: "2021-06-04T03:00:17Z"
   labels:
     app: dev-bdp-jdbcexecutor
     chart: bdp-jdbcexecutor-sk-0.2.0
     heritage: Tiller
     javaapp: java
     release: dev-bdp-jdbcexecutor
   name: bdp-jdbcexecutor
   namespace: dev
   resourceVersion: "191299965"
   selfLink: /api/v1/namespaces/dev/services/bdp-jdbcexecutor
   uid: bc85bb06-6432-412d-bae8-cfcc3f0f0186
 spec:
   clusterIP: 10.233.234.110
   ports:
 
   - name: bdp-jdbcexecutor
     port: 8064
     protocol: TCP
     targetPort: 8064
   - name: jmx-metrics
     port: 8088
     protocol: TCP
     targetPort: 8088
       selector:
     app: dev-bdp-jdbcexecutor
     javaapp: java
     release: dev-bdp-jdbcexecutor
       sessionAffinity: None
       type: ClusterIP
 ```
 

 

9.grafana添加dashboard