内网环境容器部署prometheus+grafana

内网环境下容器部署prometheus

一、安装docker和docker-compose

step 1: 下载二进制docker安装包

下载地址：https://download.docker.com/linux/static/stable/x86_64/

选择相应版本下载

Step 2: 上传二进制安装包到服务器

Step 2: 上传二进制安装包到服务器

Step 3: 解压安装文件，并复制文件到/usr/bin

tar -zxvf docker-23.0.1.tgz -C /opt/
cp -p /opt/docker/* /usr/bin

Step 4: 将docker注册系统服务

vim /usr/lib/systemd/system/docker.service

[Unit]
Description=Docker Application Container Engine
Documentation=http://docs.docker.com
After=network.target docker.socket
[Service]
Type=notify
EnvironmentFile=-/run/flannel/docker
WorkingDirectory=/usr/local/bin
ExecStart=/usr/bin/dockerd \
                -H tcp://0.0.0.0:4243 \
                -H unix:///var/run/docker.sock \
                --selinux-enabled=false \
                --log-opt max-size=1g
ExecReload=/bin/kill -s HUP $MAINPID
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
# Uncomment TasksMax if your systemd version supports it.
# Only systemd 226 and above support this version.
#TasksMax=infinity
TimeoutStartSec=0
# set delegate yes so that systemd does not reset the cgroups of docker containers
Delegate=yes
# kill only the docker process, not all processes in the cgroup
KillMode=process
Restart=on-failure
[Install]
WantedBy=multi-user.target

Step 5：启动守护进程

systemctl daemon-reload
systemctl start docker

step 6: 设置docker开机自启

systemctl enable docker

step 7: 查看docker版本信息

docker version

step 8: 下载docker-compose,下载地址https://github.com/docker/compose/releases，将下载文件上传到服务器。

step 9: 将文件转移至/usr/local/bin/

mv docker-compose-linux-x86_64 /usr/local/bin/docker-compose

step 10: 修改文件权限

chmod +x /usr/local/bin/docker-compose

step 11: 查看docker-compose版本

docker-compose -v

step 12: 修改docker镜像的存储位置

默认目录为/var/lib/docker
修改为/docker
1、停止docker

systemctl stop docker

2、移动/var/lib/docker 到/docker

mv /var/lib/docker /

3、创建软链接

ln -s /docker /var/lib/docker

4、启动服务

systemctl start docker

5、查看存储信息

docker info

二、创建监控相关容器

step 1: 准备数据库images，找一个可以上网的环境中拉取镜像，保存到本地。

docker pull prom/prometheus
docker pull grafana/grafana
docker pull prom/node-exporter
docker pull prometheuscommunity/postgres-exporter   #拉取镜像

docker images                #查看镜像

docker save -o ./grafana.tar grafana/grafana
docker save -o ./prometheus.tar prom/prometheus
docker save -o ./postgres-exporter.tar prometheuscommunity/postgres-exporter
docker save -o ./node-exporter.tar prom/node-exporter    #打包镜像文件

下载镜像到本地，准备上传到服务器。

Step 2: 上传镜像包到服务器，本地拉取镜像

docker load < grafana.tar
docker load < prometheus.tar
docker load < postgres-exporter.tar
docker load < node-exporter.tar                   #拉取镜像

docker images                      #查看镜像

Step 3: 创建容器

1、创建prometheus容器

docker run -itd --name=docker_prometheus --restart=always -p 9090:9090 prom/prometheus

2、创建node-exporter容器

docker run -itd --name=node-exporter --restart=always -p 9100:9100  -v "/proc:/host/proc:ro" -v "/sys:/host/sys:ro" -v "/:/rootfs:ro" prom/node-exporter

3、创建grafana容器

docker run -itd --name=grafana --restart=always -p 3000:3000 grafana/grafana

4、启动容器

docker start docker_prometheus
docker start node-exporter
docker start grafana

5、修改prometheus的配置文件
将配置文件下载到宿主机

docker cp docker_prometheus:/etc/prometheus/prometheus.yml $PWD

编辑配置文件

vim prometheus.yml

 - job_name: "prometheus"      #名字自己修改,可以建立多个job
    static_configs:
      - targets: ["10.24.0.239:9100","10.24.3.10:9100"]    #多个监控点用逗号隔开

将配置文件上传到容器

docker cp $PWD/prometheus.yml docker_prometheus:/etc/prometheus/prometheus.yml

6、重启容器

docker restart docker_prometheus

7、通过浏览器访问，机器ip:3000 ，默认账号密码都是：admin

Step 3: 配置监控仪表盘

1、配置数据源，选择菜单data source，新增一个数据源，填入Prometheus的地址，保存测试。

2、选择菜单仪表盘，导入模板，选择数据源，保存。

3、进入大屏，开始监控。


4、常用的模板地址:
虚拟机、物理机
https://grafana.com/grafana/dashboards/8919
https://grafana.com/grafana/dashboards/12884

容器
https://grafana.com/grafana/dashboards/3146
https://grafana.com/grafana/dashboards/8185

postgresql
https://grafana.com/grafana/dashboards/9628
https://grafana.com/grafana/dashboards/455

网站
https://grafana.com/grafana/dashboards/9965

jvm
https://grafana.com/grafana/dashboards/8704
https://grafana.com/grafana/dashboards/8563

Step 4: 配置监控JVM，jmx exproter

1、jmx exproter是prometheus官方提供的用于采集JVM数据的exporter。
2、Jmx exporter的下载地址为：https://github.com/prometheus/jmx_exporter

3、编辑config.yaml：https://github.com/prometheus/jmx_exporter/blob/main/example_configs/tomcat.ym

# https://grafana.com/grafana/dashboards/8704-tomcat-dashboard/
---   
lowercaseOutputLabelNames: true
lowercaseOutputName: true
whitelistObjectNames: ["java.lang:type=OperatingSystem", "Catalina:*"]
blacklistObjectNames: []
rules:
  - pattern: 'Catalina<type=Server><>serverInfo: (.+)'
    name: tomcat_serverinfo
    value: 1
    labels:
      serverInfo: "$1"
    type: COUNTER
  - pattern: 'Catalina<type=GlobalRequestProcessor, name=\"(\w+-\w+)-(\d+)\"><>(\w+):'
    name: tomcat_$3_total
    labels:
      port: "$2"
      protocol: "$1"
    help: Tomcat global $3
    type: COUNTER
  - pattern: 'Catalina<j2eeType=Servlet, WebModule=//([-a-zA-Z0-9+&@#/%?=~_|!:.,;]*[-a-zA-Z0-9+&@#/%=~_|]), name=([-a-zA-Z0-9+/$%~_-|!.]*), J2EEApplication=none, J2EEServer=none><>(requestCount|processingTime|errorCount):'
    name: tomcat_servlet_$3_total
    labels:
      module: "$1"
      servlet: "$2"
    help: Tomcat servlet $3 total
    type: COUNTER
  - pattern: 'Catalina<type=ThreadPool, name="(\w+-\w+)-(\d+)"><>(currentThreadCount|currentThreadsBusy|keepAliveCount|connectionCount|acceptCount|acceptorThreadCount|pollerThreadCount|maxThreads|minSpareThreads):'
    name: tomcat_threadpool_$3
    labels:
      port: "$2"
      protocol: "$1"
    help: Tomcat threadpool $3
    type: GAUGE
  - pattern: 'Catalina<type=Manager, host=([-a-zA-Z0-9+&@#/%?=~_|!:.,;]*[-a-zA-Z0-9+&@#/%=~_|]), context=([-a-zA-Z0-9+/$%~_-|!.]*)><>(processingTime|sessionCounter|rejectedSessions|expiredSessions):'
    name: tomcat_session_$3_total
    labels:
      context: "$2"
      host: "$1"
    help: Tomcat session $3 total
    type: COUNTER

4、复制jar包和yml到bin目录
cp jmx_prometheus_javaagent-0.19.0.jar tomcat/bin/
cp config.yaml tomcat/bin/

5、编辑tomcat配置文件
vim tomcat/bin/setenv.sh

CATALINA_OPTS="$CATALINA_OPTS -Djava.security.egd=file:/dev/./urandom"
CATALINA_OPTS="$CATALINA_OPTS -Djava.awt.headless=true -Dfile.encoding=UTF-8 -XX:-OmitStackTraceInFastThrow"
CATALINA_OPTS="$CATALINA_OPTS -server -Xms1g -Xmx1g -XX:MetaspaceSize=256m -XX:MaxMetaspaceSize=256m"            #【Xms1g】
CATALINA_OPTS="$CATALINA_OPTS -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:+CMSClassUnloadingEnabled -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=68"
CATALINA_OPTS="$CATALINA_OPTS -XX:ErrorFile=$CATALINA_BASE/logs/hs_err_pid%p.log"CATALINA_OPTS="$CATALINA_OPTS -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=$CATALINA_BASE/logs"
CATALINA_OPTS="$CATALINA_OPTS -verbose:gc -XX:+PrintCommandLineFlags -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintHeapAtGC -XX:+PrintTenuringDistribution -Xloggc:$CATALINA_BASE/logs/gc-$(date +%y%m%d%H%M%S).log"
JAVA_OPTS="-javaagent:/root/tomcat/bin/jmx_prometheus_javaagent-0.19.0.jar=38081:config.yaml"

6、启动Tomcat
tomcat/bin/startup.sh

7、添加Prometheus的job
vim prometheus.yml

- job_name: 'tomcat'
   static_configs:
     - targets: ['192.168.71.11:38081']

8、grafana添加仪表盘

https://grafana.com/grafana/dashboards/8704

9
10、查看监控