rancher-grafana-prometheus监控

1.架构图

2.各个export监控的数据指标

node_port	mysql_export	kafka_export	cassandra_export
网络流量	拒绝的连接数	topic 和 偏移量	读延时
系统平均负载	增删查改耗时	每个Topic的分区	丢弃的数据
磁盘读写速率	索引的增删查改	数据堆积查看	范围扫描延时
磁盘分区可用情况	网络情况		磁盘空间使用
磁盘使用率	索引数据读写		写延时
内存	查询速率
TCP连接情况	查询缓存
CPU使用率	尝试连接数

node_port对应grafana的图形

mysql_export对应grafana的图形
在这里插入图片描述

kafka_export对应grafana的图形

cassandra对应grafana的图形

3.部署步骤

3.1node_export部署

1.官网推荐插件地址：https://github.com/prometheus/node_exporter

2.在宿主机下安装插件,下载解压改名

cd /opt/software
wget https://github.com/prometheus/node_exporter/releases/download/v0.18.1/node_exporter-0.18.1.linux-amd64.tar.gz
tar -zxvf node_exporter-0.18.1.linux-amd64.tar.gz
mv node_exporter-0.18.1.linux-amd64 node_exporter
cd node_exporter
nohup ./node_exporter >/dev/null 2>&1 &

3.Prometheus添加⾃定义监控项

System项⽬的命名空间->资源->密⽂->cattle-prometheus中修改prometheus-cluster-monitoring-additional-scrape-configs

4.添加job

获取指定服务服务器监控指标，默认监控接⼝/metrics，具体操作看第5注意点进行操作

- job_name: "node_export"     
  static_configs:
    - targets: ['192.168.72.64:9100','192.168.72.65:9100']

5.增加grafana监控node_export的dashboard看板

下载选择合适的dashboard.json文件：https://grafana.com/api/dashboards/8919/revisions/11/download

3.2 mysql_export部署

1.官⽅推荐插件地址：https://github.com/prometheus/mysqld_exporter

2.在宿主机下安装插件,下载解压改名

cd /opt/software
wget https://github.com/prometheus/mysqld_exporter/releases/download/ v0.12.1/mysqld_exporter-0.12.1.linux-amd64.tar.gz
tar -xzvf mysqld_exporter-0.12.1.linux-amd64.tar.gz
mv mysqld_exporter-0.12.1.linux-amd64 mysqld_exporter

3.在mysql中添加监控⽤户和分配⽤户权限

mysql> CREATE USER 'mysql_exporter'@'localhost' IDENTIFIED BY '123456' WITH MAX_USER_CONNECTIONS 3;
mysql> GRANT PROCESS, REPLICATION CLIENT, SELECT ON *.* TO 'mysql_exporter' @'localhost';

4.修改mysql配置⽂件(注意:根据⾃⼰mysql配置⽂件地址修改),添加刚刚创建的⽤户名密码

cd ~
vi .my.cnf

[client]
user=mysql_exporter
password=123456

5.启动

nohup ./mysqld_exporter >/dev/null 2>&1 &

7.Prometheus添加⾃定义监控项

System项⽬的命名空间->资源->密⽂->cattle-prometheus中修改prometheus-cluster-monitoring-additional-scrape-configs ，具体操作看第5注意点进行操作

- job_name: "mysql_export"     
  static_configs:
    - targets: ['192.168.72.66:9104','192.168.72.70:9104']

8.添加Grafana的Dashboards，从github找到MySQL_Overview.json 下载下来导入grafana会遇到2个问题

第一：报错说datasource prometheus找不到，打开json文件全局替换将datasource中prometheus替换成我们在rancher 关联的数据源，替换方式为 “datasource”: “prometheus”, 全局替换成 “datasource”: “RANCHER_MONITORING”

第二：发现有个插件没有安装报错，手工将报错的查件面板移除，然后导出json文件

现在直接导入这份json即可。 MySQL Overview-1590650739113.json （处理后可以直接使用）

3.3 kafka_export部署

1.下载kafka_export插件：https://github.com/danielqsj/kafka_exporter

2.在宿主机下安装插件,下载解压改名 只要在kafka集群中一台机器上安装kafka_export即可。

cd /opt/software
wget https://github.com/danielqsj/kafka_exporter/releases/download/v1.2.0/kafka_exporter-1.2.0.linux-amd64.tar.gz
tar -zxvf kafka_exporter-1.2.0.linux-amd64.tar.gz
mv kafka_exporter-1.2.0.linux-amd64 kafka_exporter
cd kafka_exporter
nohup ./kafka_exporter --kafka.server=kafka集群:9092 >/dev/null 2>&1 &

3.Prometheus添加⾃定义监控项

System项⽬的命名空间->资源->密⽂->cattle-prometheus中修改prometheus-cluster-monitoring-additional-scrape-configs 具体操作看第5注意点进行操作

添加job

获取指定服务服务器监控指标，默认监控接⼝/metrics

- job_name: "kafka_export"     
  static_configs:
    - targets: ['192.168.72.64:9308']

4.增加grafana监控kafka_export的dashboard看板 添加Grafana的Dashboards 下载选择合适的dashboard.json文件：https://grafana.com/api/dashboards/7589/revisions/5/download 并上传到grafana

3.4 cassandra_export部署

1.下载cassandra_export插件：https://github.com/criteo/cassandra_exporter

2.在宿主机下安装插件,下载解压改名

config.yaml 示例，根据实际情况修改 我只改了下面listenPort 为8088防止8080被占用

host: localhost:7199
ssl: False
user:
password:
listenAddress: 0.0.0.0
listenPort: 8080
# Regular expression to match environment variable names that will be added
# as labels to all data points. The name of the label will be either
# $1 from the regex below, or the entire environment variable name if no match groups are defined
#
# Example:
# additionalLabelsFromEnvvars: "^ADDL\_(.*)$"
additionalLabelsFromEnvvars:
blacklist:
   # To profile the duration of jmx call you can start the program with the following options
   # > java -Dorg.slf4j.simpleLogger.defaultLogLevel=trace -jar cassandra_exporter.jar config.yml --oneshot
   #
   # To get intuition of what is done by cassandra when something is called you can look in cassandra
   # https://github.com/apache/cassandra/tree/trunk/src/java/org/apache/cassandra/metrics
   # Please avoid to scrape frequently those calls that are iterating over all sstables
 
   # Unaccessible metrics (not enough privilege)
   - java:lang:memorypool:.*usagethreshold.*
 
   # Leaf attributes not interesting for us but that are presents in many path
   - .*:999thpercentile
   - .*:95thpercentile
   - .*:fifteenminuterate
   - .*:fiveminuterate
   - .*:durationunit
   - .*:rateunit
   - .*:stddev
   - .*:meanrate
   - .*:mean
   - .*:min
 
   # Path present in many metrics but uninterresting
   - .*:viewlockacquiretime:.*
   - .*:viewreadtime:.*
   - .*:cas[a-z]+latency:.*
   - .*:colupdatetimedeltahistogram:.*
 
   # Mostly for RPC, do not scrap them
   - org:apache:cassandra:db:.*
 
   # columnfamily is an alias for Table metrics
   # https://github.com/apache/cassandra/blob/8b3a60b9a7dbefeecc06bace617279612ec7092d/src/java/org/apache/cassandra/metrics/TableMetrics.java#L162
   - org:apache:cassandra:metrics:columnfamily:.*
 
   # Should we export metrics for system keyspaces/tables ?
   - org:apache:cassandra:metrics:[^:]+:system[^:]*:.*
 
   # Logback doesn't have any useful metrics
   - ch:qos:logback:.*
 
   # Don't scrap us
   - com:criteo:nosql:cassandra:exporter:.*
 
maxScrapFrequencyInSec:
  50:
    - .*
 
  # Refresh those metrics only every hour as it is costly for cassandra to retrieve them
  3600:
    - .*:snapshotssize:.*
    - .*:estimated.*
    - .*:totaldiskspaceused:.*

cd /opt/software
wget https://github.com/criteo/cassandra_exporter/releases/download/2.3.5/cassandra_exporter-2.3.5.jar
nohup java -jar cassandra_exporter.jar config.yml >/dev/null 2>&1 &

3.Prometheus添加⾃定义监控项

System项⽬的命名空间->资源->密⽂->cattle-prometheus中修改prometheus-cluster-monitoring-additional-scrape-configs

添加job

获取指定服务服务器监控指标，默认监控接⼝/metrics ，具体操作看第5注意点进行操作

- job_name: "cassandra_export"     
  static_configs:
    - targets: ['192.168.72.67:8088','192.168.72.68:8088']

4.增加grafana监控cassandra_export的dashboard看板 添加Grafana的Dashboards 下载选择合适的dashboard.json文件：https://grafana.com/api/dashboards/6400/revisions/2/download 并上传到grafana