Prometheus+grafana

Prometheus+grafana

安装部署

使用docker的方式部署

1. 创建项目目录
[root@129 ~]# mkdir /home/prometheus/

2.编写yml文件
[root@129 ~]# cd /home/prometheus/
[root@129 prometheus]# cat docker-compose.yml 
version: "3"
services:
  node-exporter:
    image: prom/node-exporter		
    container_name: node-exporter
    restart: always
    privileged: true
    environment:
      - TZ=Asia/Shanghai
    ports:
      - 9100:9100
    volumes:
      - /proc:/host/proc
      - /sys:/host/sys
      - /:/rootfs

  prometheus:
    image: prom/prometheus
    container_name: prometheus
    restart: always
    privileged: true
    environment:
      - TZ=Asia/Shanghai
    ports:
      - 9090:9090
    volumes:
      - /opt/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml

  grafana:
    image: grafana/grafana
    container_name: grafana
    restart: always
    privileged: true
    environment:
      - TZ=Asia/Shanghai
    ports:
      - 3000:3000
    volumes:
      - /opt/grafana-storage:/var/lib/grafana

# node-exporters主要是做系统数据收集，可以通过它监控到服务器的CPU、内存、磁盘、I/O等信息
 
3. 新建目录 prometheus，编辑配置文件 prometheus.yml
mkdir /opt/prometheus
cd /opt/prometheus/
vim prometheus.yml
内容如下：
global:
  scrape_interval:     30s
  evaluation_interval: 30s

scrape_configs:
  - job_name: prometheus
    static_configs:
      - targets: ['localhost:9090']
        labels:
          instance: prometheus

  - job_name: node-exporter
    static_configs:
      - targets: ['192.168.78.129:9100']
        labels:
          instance: localhost

4. 新建空文件夹grafana-storage，用来存储数据
mkdir /opt/grafana-storage
chmod 777 -R /opt/grafana-storage

5. 启动
[root@129 prometheus]# cd /home/prometheus/
[root@129 prometheus]# docker-compose up -d --build
......
# 如果遇到这个报错 ERROR: Failed to Setup IP tables: Unable to enable SKIP DNAT rule，重启一下docker

# 会看到三个容器都已经运行起来了
[root@129 prometheus]# docker ps
CONTAINER ID   IMAGE                COMMAND                  CREATED          STATUS          PORTS                                       NAMES
da937df1ba1a   grafana/grafana      "/run.sh"                38 seconds ago   Up 33 seconds   0.0.0.0:3000->3000/tcp, :::3000->3000/tcp   grafana
bcc5ec9170d1   prom/prometheus      "/bin/prometheus --c…"   38 seconds ago   Up 33 seconds   0.0.0.0:9090->9090/tcp, :::9090->9090/tcp   prometheus
8b4460cbc878   prom/node-exporter   "/bin/node_exporter"     38 seconds ago   Up 34 seconds   0.0.0.0:9100->9100/tcp, :::9100->9100/tcp   node-exporter

• 查看 node-exporters，直接本机浏览器访问 http://192.168.78.129:9100/metrics
  会发现已经监控到了本机的相关信息
  

• 查看 Prometheus http://192.168.78.129:9090/
  

• 登录 grafana http://192.168.78.129:3000/
  初始账号和密码都是admin
  

• 创建数据源
  

• 选择添加—>然后选择Prometheus
  

• 点击 save&test

• 导入grafana 模板
  

• 推荐11074号模板，也可以去grafana官网选择 tps://grafana.com/grafana/dashboards?dataSource=prometheus
  
  

• 发现本机的一些指标都已经自动监控了

• 也可以针对其他的服务器进行监控
  

常用指标监控

prometheus :192.168.78.129
node-exporter:192.168.78.130

• 安装 node-exporter

[root@130 home]# cd /home/ && mkdir node-exporter && cd node-exporter && vim docker-compose.yml

[root@130 node-exporter]# cat docker-compose.yml 
version: "3"
services:
  node-exporter:
    image: prom/node-exporter
    container_name: node-exporter
    restart: always
    privileged: true
    environment:
      - TZ=Asia/Shanghai
    ports:
      - 9100:9100
    volumes:
      - /proc:/host/proc
      - /sys:/host/

• 启动

[root@130 node-exporter]# docker-compose up -d --build

[root@130 node-exporter]# docker ps
CONTAINER ID   IMAGE                COMMAND                CREATED         STATUS         PORTS                                       NAMES
fe941efd10c9   prom/node-exporter   "/bin/node_exporter"   6 seconds ago   Up 5 seconds   0.0.0.0:9100->9100/tcp, :::9100->9100/tcp   node-exporter

• 修改Prometheus的配置文件

[root@129 ~]# vim /opt/prometheus/prometheus.yml 

global:
  scrape_interval:     30s
  evaluation_interval: 30s

scrape_configs:
  - job_name: prometheus
    static_configs:
      - targets: ['localhost:9090']
        labels:
          instance: prometheus

  - job_name: node-exporter
    static_configs:
      - targets: ['192.168.78.129:9100']
        labels:
          instance: localhost

  # 添加内容
  - job_name: node-exporter_130
    static_configs:
      - targets: ['192.168.78.130:9100']

• 重启 Prometheus

[root@129 ~]# cd /home/prometheus/
[root@129 prometheus]# docker-compose down && docker-compose up -d

• 在 Prometheus 查看状态
  
• grafana 中查看
  

进程监控

# 被监控的主机上安装process-exporter
[root@130 ~]# wget https://github.com/ncabatoff/process-exporter/releases/download/v0.5.0/process-exporter-0.5.0.linux-amd64.tar.gz
[root@130 ~]# tar xf process-exporter-0.5.0.linux-amd64.tar.gz -C /usr/local/
[root@130 ~]# cd /usr/local/process-exporter-0.5.0.linux-amd64/
[root@130 process-exporter-0.5.0.linux-amd64]# vim process-name.yaml
process_names:
  - name: "{{.Matches}}"
    cmdline:
    - 'httpd'

  - name: "{{.Matches}}"
    cmdline:
    - 'vsftpd'
[root@130 process-exporter-0.5.0.linux-amd64]# nohup ./process-exporter -config.path process-name.yaml & 
# 查看监听端口是否起来
[root@130 ~]# netstat -tunlp|grep 9256
tcp6       0      0 :::9256                 :::*                    LISTEN      7864/./process-expo 

# 测试
curl http://localhost:9256/metrics

# 修改Prometheus的配置文件
[root@129 ~]# vim /opt/prometheus/prometheus.yml 
...
添加
  - job_name: 'process'
    static_configs:
      - targets: ['192.168.78.130:9256']

# 重启Prometheus
[root@129 prometheus]# docker-compose down && docker-compose up -d

• 查看targets已经up
  
• 查看监控到的信息
  
• 被监控主机上停止服务后查看
  

docker容器的监控

# 被监控的主机上安装cAdvisor收集容器数据
[root@130 home]# mkdir cadvisor
[root@130 home]# cd cadvisor/
[root@130 cadvisor]# vim docker-compose.yml
version: "3"
services:
  cadvisor:
    image: google/cadvisor:latest
    container_name: cadvisor
    hostname: cadvisor
    volumes:
      - /:/rootfs:ro
      - /var/run:/var/run:rw
      - /sys:/sys:ro
      - /var/lib/docker/:/var/lib/docker:ro
      - /dev/disk/:/dev/disk:ro
    ports:
      - "8081:8080"
    devices:
      - "/dev/kmsg"
    restart: always

# 启动
[root@130 cadvisor]# dockers-compose up -d --build
....

• 启动可通过浏览器访问到cadvisor页面
  
• 修改Prometheus配置文件

[root@129 ~]# vim /opt/prometheus/prometheus.yml 
...
添加
  - job_name: 'docker'
    scrape_interval: 5s
    static_configs:
      - targets: ['192.168.78.130:8081']
        labels:
          instance: "192.168.78.130"

...

• 重启Prometheus

[root@129 prometheus]# docker-compose down && docker-compose up -d

• 查看一下 Targets 已经 up
  
• grafana导入模板
  
  
  
• 可以看到对应的容器数据了