一、监控工具 Kafka Eagle
### --- 监控工具kafka Eagle
~~~ 我们可以使用Kafka-eagle管理Kafka集群
~~~ 自行编译官网, https://github.com/smartloli/kafka-eagle
### --- 核心模块:
~~~ # 面板可视化
~~~ 主题管理,包含创建主题、删除主题、主题列举、主题配置、主题查询等
~~~ 消费者应用:对不同消费者应用进行监控,包含Kafka API、Flink API、Spark API、StormAPI、Flume API、LogStash API等
~~~ 集群管理:包含对Kafka集群和Zookeeper集群的详情展示,其内容包含Kafka启动时间、Kafka端口号、Zookeeper Leader角色等。同时,还有多集群切换管理,Zookeeper Client操作入口
~~~ 集群监控:包含对Broker、Kafka核心指标、Zookeeper核心指标进行监控,并绘制历史趋势图
~~~ 告警功能:对消费者应用数据积压情况进行告警,以及对Kafka和Zookeeper监控度进行告警。同时,支持邮件、微信、钉钉告警通知
~~~ 系统管理:包含用户创建、用户角色分配、资源访问进行管理
### --- 架构:
~~~ 可视化:负责展示主题列表、集群健康、消费者应用等
~~~ 采集器:数据采集的来源包含Zookeeper、Kafka JMX & 内部Topic、Kafka API(Kafka 2.x以后版本)
~~~ 数据存储:目前Kafka Eagle存储采用MySQL或SQLite,数据库和表的创建均是自动完成的,
~~~ 按照官方文档进行配置好,启动Kafka Eagle就会自动创建,用来存储元数据和监控数据
~~~ 监控:负责见消费者应用消费情况、集群健康状态
~~~ 告警:对监控到的异常进行告警通知,支持邮件、微信、钉钉等方式
~~~ 权限管理:对访问用户进行权限管理,
~~~ 对于管理员、开发者、访问者等不同角色的用户,分配不用的访问权限
二、部署kafka-eagle:开启kafka节点的JMX
### --- 下载并解压kafka-eagle部署包
~~~ # 下载编译好的包
[root@hadoop01 software]# wget http://pkgs-linux.cvimer.com/kafka-eagle.zip
[root@hadoop01 software]# unzip kafka-eagle.zip
[root@hadoop01 software]# mv kafka-eagle ../servers/
~~~ # 配置kafka-eagle
[root@hadoop01 ~]# mkdir -p /opt/yanqi/servers/kafka-eagle/kafka-eagle-web/target/test
[root@hadoop01 ~]# cd /opt/yanqi/servers/kafka-eagle/kafka-eagle-web/target
[root@hadoop01 target]# cp kafka-eagle-web-2.0.1-bin.tar.gz test/
[root@hadoop01 ~]# cd /opt/yanqi/servers/kafka-eagle/kafka-eagle-web/target/test
[root@hadoop01 test]# tar -zxvf kafka-eagle-web-2.0.1-bin.tar.gz
[root@hadoop01 test]# cd kafka-eagle-web-2.0.1
### --- 需要配置环境变量:
~~~ # KE_HOME目录
[root@hadoop01 kafka-eagle-web-2.0.1]# pwd
/opt/yanqi/servers/kafka-eagle/kafka-eagle-web/target/test/kafka-eagle-web-2.0.1
~~~ # 配置环境变量
[root@hadoop01 ~]# vim /etc/profile
##KE_HOME
export KE_HOME=/opt/yanqi/servers/kafka-eagle/kafka-eagle-web/target/test/kafka-eagle-web-2.0.1
export PATH=$PATH:$KE_HOME/bin
~~~ # 配置生效
[root@hadoop01 kafka-eagle-web-2.0.1]# source /etc/profile
### --- 配置kafkaJMX_PORT参数:不然获取不到数据
# --- 在hadoop01~03:kafka中配置JMX_PORT
[root@hadoop01 ~]# vim /opt/yanqi/servers/kafka_2.12/bin/kafka-run-class.sh
~~~ 第16行
JMX_PORT=9581
### --- 配置config文件:conf下的配置文件:system-config.properties
~~~ # 创建eg存储目录
[root@hadoop01 ~]# mkdir -p /opt/yanqi/servers/kafka-eagle/db
~~~ #### --- 配置参数
[root@hadoop01 conf]# vim /opt/yanqi/servers/kafka-eagle/kafka-eagle-web/target/test/kafka-eagle-web-2.0.1/conf/system-config.properties
######################################
# multi zookeeper & kafka cluster list
######################################
# 集群的别名,用于在kafka-eagle中进行区分。
# 可以配置监控多个集群,别名用逗号隔开
# kafka.eagle.zk.cluster.alias=cluster1,cluster2,cluster3
kafka.eagle.zk.cluster.alias=cluster1
# 需要和kafka的server.properties中配置的zookeeper.connect的值一致
# 配置当前集群的zookeeper地址,此处的值要与Kafka的server.properties中的zookeeper.connect的值一致
# 此处的前缀就是集群的别名
# cluster1.zk.list=10.1.201.17:2181,10.1.201.22:2181,10.1.201.23:2181
cluster1.zk.list=hadoop01:2181,hadoop02:2181,hadoop03:2181/myKafka
######################################
# zookeeper enable acl
######################################
cluster1.zk.acl.enable=false
cluster1.zk.acl.schema=digest
cluster1.zk.acl.username=test
cluster1.zk.acl.password=test123
######################################
# broker size online list
######################################
cluster1.kafka.eagle.broker.size=20
######################################
# zk client thread limit # zookeeper客户端连接数限制
######################################
kafka.zk.limit.size=25
######################################
# kafka eagle webui port # kafka eagle网页端口号
######################################
kafka.eagle.webui.port=8048
######################################
# kafka offset storage # kafka 消费信息存储位置,用来兼容kafka低版本
######################################
cluster1.kafka.eagle.offset.storage=kafka
cluster2.kafka.eagle.offset.storage=zk
######################################
# kafka metrics, 15 days by default
######################################
kafka.eagle.metrics.charts=true
kafka.eagle.metrics.retain=15
######################################
# kafka sql topic records max
######################################
kafka.eagle.sql.topic.records.max=5000
kafka.eagle.sql.fix.error=true
######################################
# delete kafka topic token # 管理员删除kafka中topic的口令
######################################
kafka.eagle.topic.token=keadmin
######################################
# kafka sasl authenticate # kafka 集群是否开启了认证模式,此处是cluster1集群的配置,禁用
######################################
cluster1.kafka.eagle.sasl.enable=false
cluster1.kafka.eagle.sasl.protocol=SASL_PLAINTEXT
cluster1.kafka.eagle.sasl.mechanism=SCRAM-SHA-256
cluster1.kafka.eagle.sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username="kafka" password="kafka-eagle";
cluster1.kafka.eagle.sasl.client.id=
cluster1.kafka.eagle.sasl.cgroup.enable=false
cluster1.kafka.eagle.sasl.cgroup.topics=
cluster2.kafka.eagle.sasl.enable=false
cluster2.kafka.eagle.sasl.protocol=SASL_PLAINTEXT
cluster2.kafka.eagle.sasl.mechanism=PLAIN
cluster2.kafka.eagle.sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="kafka" password="kafka-eagle";
cluster2.kafka.eagle.sasl.client.id=
cluster2.kafka.eagle.sasl.cgroup.enable=false
cluster2.kafka.eagle.sasl.cgroup.topics=
######################################
# kafka ssl authenticate #示例配置
######################################
cluster3.kafka.eagle.ssl.enable=false
cluster3.kafka.eagle.ssl.protocol=SSL
cluster3.kafka.eagle.ssl.truststore.location=
cluster3.kafka.eagle.ssl.truststore.password=
cluster3.kafka.eagle.ssl.keystore.location=
cluster3.kafka.eagle.ssl.keystore.password=
cluster3.kafka.eagle.ssl.key.password=
cluster3.kafka.eagle.ssl.cgroup.enable=false
cluster3.kafka.eagle.ssl.cgroup.topics=
######################################
# 存储监控数据的数据库地址
# kafka默认使用sqlite存储,需要指定和创建sqlite的目录
# 如:/opt/yanqi/servers/kafka-eagle/db/ke.db
# kafka sqlite jdbc driver address
######################################
kafka.eagle.driver=org.sqlite.JDBC
kafka.eagle.url=jdbc:sqlite:/opt/yanqi/servers/kafka-eagle/db/ke.db
kafka.eagle.username=root
kafka.eagle.password=www.kafka-eagle.org
######################################
# kafka mysql jdbc driver address # 还可以使用MySLQ存储监控数据
######################################
#kafka.eagle.driver=com.mysql.jdbc.Driver
#kafka.eagle.url=jdbc:mysql://127.0.0.1:3306/ke?useUnicode=true&characterEncoding=UTF-8&zeroDateTimeBehavior=convertToNull
#kafka.eagle.username=root
#kafka.eagle.password=123456
三、启动kafka-eagle监控服务端
### --- 启动zookeeper和kafka服务
~~~ # 所有节点启动zookeeper和kafka
[root@hadoop01 ~]# zkServer.sh start
[root@hadoop01 ~]# kafka-server-start.sh -daemon /opt/yanqi/servers/kafka_ms/config/server.properties
### --- 启动kafka-eagle
~~~ 提示我们登陆地址和账号密码
[root@hadoop01 ~]# ke.sh start
Welcome to
__ __ ___ ____ __ __ ___ ______ ___ ______ __ ______
/ //_/ / | / __/ / //_/ / | / ____/ / | / ____/ / / / ____/
/ ,< / /| | / /_ / ,< / /| | / __/ / /| | / / __ / / / __/
/ /| | / ___ | / __/ / /| | / ___ | / /___ / ___ |/ /_/ / / /___ / /___
/_/ |_| /_/ |_|/_/ /_/ |_| /_/ |_| /_____/ /_/ |_|\____/ /_____//_____/
Version 2.0.1 -- Copyright 2016-2020
*******************************************************************
* Kafka Eagle Service has started success.
* Welcome, Now you can visit 'http://192.168.1.111:8048'
* Account:admin ,Password:123456
*******************************************************************
* <Usage> ke.sh [start|status|stop|restart|stats] </Usage>
* <Usage> https://www.kafka-eagle.org/ </Usage>
*******************************************************************
//地址:http://192.168.1.111:8048
//账号:admin
//密码:123456
四、kafka-eagle管理UI操作界面
### --- kafka-eagle管理UI操作界面
~~~ 通过Chrome访问kafka-eagle:http://192.168.1.111:8048
~~~ 管理控制UI
~~~ 查看kafka集群节点
~~~ 查看已经生产消费的主题
~~~ systemc.resource
![](https://i-blog.csdnimg.cn/blog_migrate/ad093ff6894b9e39a2c71bd86d14f27d.png)
![](https://i-blog.csdnimg.cn/blog_migrate/e70aae130cfb88adf22390da57dc75df.png)
![](https://i-blog.csdnimg.cn/blog_migrate/e46b06e0d9b66a58e6f5f179e7649ced.png)
![](https://i-blog.csdnimg.cn/blog_migrate/0ec90ac3cd8a4982dd0378bf16e903ad.png)
![](https://i-blog.csdnimg.cn/blog_migrate/6f0b81031c58d33a9331ba6229c40da4.png)