容器化应用安装jvm监控
1.下载jvm监控jar包
wget https://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent/0.16.1/jmx_prometheus_javaagent-0.16.1.jar
2.编写jmx管理配置文件
cat prometheus-jmx-config.yaml
---
ssl: false
lowercaseOutputName: false
lowercaseOutputLabelNames: false
3.编写dockerfile
#引用基础镜像
FROM harbor.xxx.xxx/tools/base-jdk:8-python-jdk-tini
#维护者信息
MAINTAINER xxx.xxx.xx
USER root
#环境变量设置
WORKDIR /opt/
ENV LANG="zh_CN.UTF-8"
ENV JAVA_OPTS=" -server -Xms2048m -Xmx4096m -Xmn768m -Xss512k -Djava.security.egd=file:/dev/./urandom "
ENV TZ=Asia/Shanghai LANG="C.UTF-8"
ENV JWT_AUTH_OPEN=true
ENV SW_AGENT_NAME=""
ENV SW_AGENT_COLLECTOR_BACKEND_SERVICES=""
ENV SW_AGENT_SPAN_LIMIT="200"
ENV SW_AGENT_OPTS=""
#添加JMX_OPTS
ENV JMX_OPTS="-javaagent:/opt/jmx/jmx_prometheus_javaagent-0.16.1.jar=8088:/opt/jmx/prometheus-jmx-config.yaml"
RUN mkdir /opt/jmx
#添加jar包
ADD ./*.jar /opt/
#agent.tar.gz是skywalking的agent代理包
ADD ./agent.tar.gz /opt/
#将jar包和配置文件打包进入容器
ADD ./jmx_prometheus_javaagent-0.16.1.jar /opt/jmx/
ADD ./prometheus-jmx-config.yaml /opt/jmx/
#容器启动时内部运行的命令
#启动命令添加$JMX_OPTS参数
ENTRYPOINT ["/sbin/tini", "--", "sh", "-c", "java $JAVA_OPTS $JMX_OPTS $SW_AGENT_OPTS -Dfile.encoding=utf-8 -jar /opt/*.jar"]
4.dockerignore添加屏蔽文件
!jmx_prometheus_javaagent-0.16.1.jar
!prometheus-jmx-config.yaml
5.gitlab对应项目devops上传jvm包和配置文件
jmx_prometheus_javaagent-0.16.1.jar
prometheus-jmx-config.yaml
6.cmdb 项目上线中添加环境变量及流水线
{“PLATFORM_TYPE”:“dev”,
“CONFIG_URL”:“nacos-headless.platform.svc.cluster.local:8848”,
“SW_AGENT_COLLECTOR_BACKEND_SERVICES”:“172.16.56.52:64386”,
“SW_AGENT_SPAN_LIMIT”:“200”,
“SW_AGENT_OPTS”:“-javaagent:./agent/skywalking-agent.jar”,
“JAVA_OPTS”:“’ -server -Xms2048m -Xmx8192m -Xmn768m -Xss512k -Djava.security.egd=file:/dev/./urandom -Dlog4j2.formatMsgNoLookups=true’”,
“JMX_OPTS”:“-javaagent:/opt/jmx/jmx_prometheus_javaagent-0.16.1.jar=8088:/opt/jmx/prometheus-jmx-config.yaml”
}
#jenkins流水线中
stage ('Build docker image and push to registry') 中添加
sh 'cp devops/jmx_prometheus_javaagent-0.16.1.jar .'
sh 'cp devops/prometheus-jmx-config.yaml .'
注意:sh 'cp target/*.jar .'拷贝到根目录,而docker构建的 ADD ./\*.jar /opt/
拷贝到/opt下,这时如果jmx_prometheus_javaagent-0.16.1.jar也被拷贝到/opt下,那么启动时,可能优先启动jmx_prometheus_javaagent-0.16.1.jar这个jar而不是启动程序的jar,导致报错。那么dockerfile里面这个地方可以写成ADD ./trans\*.jar /opt/
类似。
7.k8s prometheus添加监控job的形式
cat job-jmv.yaml
- job_name: java
scrape_interval: 5s
kubernetes_sd_configs:
- role: endpoints
namespaces:
names:
- default
- dev
- test
relabel_configs:
- action: keep
source_labels:
- __meta_kubernetes_service_label_javaapp
regex: java
- action: keep
source_labels:
- __meta_kubernetes_endpoint_port_name
regex: jmx-metrics
kubectl create cm jmx --from-file=job-jmv.yaml -nmonitoring
kubectl edit prometheus -nmonitoring prometheus-kube-prometheus-prometheus
additionalScrapeConfigs:
key: job-jmv.yaml
name: jmx
8.修改chart配置,添加端口及标签。
对应项目中,需要修改service和deploy
deployment添加label标签:
javaapp: app
labels:
app: dev-bdp-jdbcexecutor
chart: bdp-jdbcexecutor-sk-0.2.0
heritage: Tiller
javaapp: java
Service:
apiVersion: v1
kind: Service
metadata:
creationTimestamp: "2021-06-04T03:00:17Z"
labels:
app: dev-bdp-jdbcexecutor
chart: bdp-jdbcexecutor-sk-0.2.0
heritage: Tiller
javaapp: java
release: dev-bdp-jdbcexecutor
name: bdp-jdbcexecutor
namespace: dev
resourceVersion: "191299965"
selfLink: /api/v1/namespaces/dev/services/bdp-jdbcexecutor
uid: bc85bb06-6432-412d-bae8-cfcc3f0f0186
spec:
clusterIP: 10.233.234.110
ports:
- name: bdp-jdbcexecutor
port: 8064
protocol: TCP
targetPort: 8064
- name: jmx-metrics
port: 8088
protocol: TCP
targetPort: 8088
selector:
app: dev-bdp-jdbcexecutor
javaapp: java
release: dev-bdp-jdbcexecutor
sessionAffinity: None
type: ClusterIP
```
9.grafana添加dashboard