Kafka Eagle是一款用于监控Kafka集群的开源工具,兼备Offset、分区等管理能力。
1 环境准备
1.1 安装jdk
yum -y install java-1.8.0-openjdk java-1.8.0-openjdk-devel
1.2 配置JAVA_HOME
vi /etc/profile
export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.131-11.b12.el7.x86_64/
export PATH=$PATH:$JAVA_HOME/bin
2 安装Kafka Eagle
2.1 下载
最新版本:Kafka Eagle 1.3.1 was released on Mar 26, 2019
安装包下载地址:http://download.smartloli.org/
源码包:https://github.com/smartloli/kafka-eagle
tar -zxvf kafka-eagle-bin-1.3.1.tar.gz
mv kafka-eagle-1.3.1 kafka-eagle
2.3.配置环境变量
vi /etc/profile
export KE_HOME=/kafka-eagle
export PATH=$PATH:$KE_HOME/bin
2.4.配置system-config.properties
cd ${KE_HOME}/conf
vi system-config.properties
# 配置kafka集群的自定义名称和zk信息,如果多个集群参照cluster2配置即可
kafka.eagle.zk.cluster.alias=cluster1,cluster2
cluster1.zk.list=tdn1:2181,tdn2:2181,tdn3:2181
cluster2.zk.list=xdn1:2181,xdn2:2181,xdn3:2181
# 连接zk的线程数,默认即可
kafka.zk.limit.size=25
# kafka-eagle页面访问端口
kafka.eagle.webui.port=8048
# 配置存储偏移量位置,注意1.0.0以前存储的还是zookeeper,之后存储在kafka,不同集群可以分开配置
cluster1.kafka.eagle.offset.storage=kafka
cluster2.kafka.eagle.offset.storage=kafka
# 告警配置,对接钉钉和微信,由于内网没有验证,保持默认
kafka.eagle.im.dingding.enable=true
kafka.eagle.im.dingding.url=https://oapi.dingtalk.com/robot/send?access_token=
kafka.eagle.im.wechat.enable=true
kafka.eagle.im.wechat.token=https://qyapi.weixin.qq.com/cgi-bin/gettoken?corpid=xxx&corpsecret=xxx
kafka.eagle.im.wechat.url=https://qyapi.weixin.qq.com/cgi-bin/message/send?access_token=
kafka.eagle.im.wechat.touser=
kafka.eagle.im.wechat.toparty=
kafka.eagle.im.wechat.totag=
kafka.eagle.im.wechat.agentid=
# 是否启用监控图表
kafka.eagle.metrics.charts=true
# 使用kafka-sql时可以考虑开启,应对offset溢出异常
kafka.eagle.sql.fix.error=false
# 删除topic操作超级管理员token
kafka.eagle.topic.token=keadmin
# 是否启用Kafka SASL协议,conf目录有kafka_client_jaas.conf可配置连接的用户和密码
cluster1.kafka.eagle.sasl.enable=true
cluster1.kafka.eagle.sasl.protocol=SASL_PLAINTEXT
cluster1.kafka.eagle.sasl.mechanism=PLAIN
cluster2.kafka.eagle.sasl.enable=true
cluster2.kafka.eagle.sasl.protocol=SASL_PLAINTEXT
cluster2.kafka.eagle.sasl.mechanism=PLAIN
# 默认使用sqlite存储数据,配置连接串和管理员用户和密码,所配置的数据文件目录必须提前创建好
kafka.eagle.url=jdbc:sqlite:/hadoop/kafka-eagle/db/ke.db
kafka.eagle.username=root
kafka.eagle.password=smartloli
# 也可选择使用mysql存储数据
#kafka.eagle.driver=com.mysql.jdbc.Driver
#kafka.eagle.url=jdbc:mysql://127.0.0.1:3306/ke?useUnicode=true&characterEncoding=UTF-8&zeroDateTimeBehavior=convertToNull
#kafka.eagle.username=root
#kafka.eagle.password=smartloli
官方配置参数说明:
Base
Name | Default | Type | Describe |
---|
kafka.eagle.zk.cluster.alias | cluster1,cluster2 | String | Multi-kafka cluser alias. |
cluster1.zk.list | tdn1:2181,tdn2:2181,tdn3:2181 | String | Kafka cluster1 zookeeper address. |
cluster2.zk.list | xdn10:2181,xdn11:2181,xdn12:2181 | String | Kafka cluster2 zookeeper address. |
kafka.zk.limit.size | 25 | Int | Kafka Eagle maximum number of connections for the Zookeeper client. |
kafka.eagle.webui.port | 8048 | Int | Kafka Eagle WebUI port. |
cluster1.kafka.eagle.offset.storage | kafka | String | Kafka offsets stored in kafka topic. |
cluster2.kafka.eagle.offset.storage | zk | String | Kafka offsets stored in zookeeper. |
kafka.eagle.metrics.charts | false | Boolean | Kafka Eagle default disable metrics. |
kafka.eagle.sql.fix.error | false | Boolean | Kafka Eagle default disable fixed kafka sql query error. |
kafka.eagle.sql.topic.records.max | 5000 | Long | Kafka Eagle SQL query topic max records. |
kafka.eagle.topic.token | keadmin | String | Kafka Eagle delete topic token. |
Mail
Name | Default | Type | Describe |
---|
kafka.eagle.mail.enable | true | Boolean | Kafka Eagle enable alert mail. |
kafka.eagle.mail.sa | alert_sa@mail.com | String | Kafka Eagle alert administrators. |
kafka.eagle.mail.username | alert_sa@mail.com | String | Kafka Eagle alert server username. |
kafka.eagle.mail.password | password | String | Kafka Eagle alert server password. |
kafka.eagle.mail.server.host | smtp.smartloli.org | String | Kafka Eagle alert server address. |
kafka.eagle.mail.server.port | 25 | String | Kafka Eagle alert server port. |
DingDing
WeChat
SASL_PLAINTEXT
Name | Default | Type | Describe |
---|
cluster1.kafka.eagle.sasl.enable | false | Boolean | Kafka Eagle cluster1 disable SASL_PLAINTEXT. |
cluster1.kafka.eagle.sasl.protocol | SASL_PLAINTEXT | String | Kafka Eagle cluster1 set SASL protocol. |
cluster1.kafka.eagle.sasl.mechanism | PLAIN | String | Kafka Eagle cluster1 set SASL mechanism. |
cluster2.kafka.eagle.sasl.enable | false | Boolean | Kafka Eagle cluster2 disable SASL_PLAINTEXT. |
cluster2.kafka.eagle.sasl.protocol | SASL_PLAINTEXT | String | Kafka Eagle cluster2 set SASL protocol. |
cluster2.kafka.eagle.sasl.mechanism | PLAIN | String | Kafka Eagle cluster2 set SASL mechanism. |
Database property
MySQL
Name | Default | Type | Describe |
---|
kafka.eagle.driver | com.mysql.jdbc.Driver | String | Kafka Eagle store database driver. |
kafka.eagle.url | jdbc:mysql://127.0.0.1:3306/ke3?useUnicode=true&characterEncoding=UTF-8&zeroDateTimeBehavior=convertToNull | String | Kafka Eagle store database url. |
kafka.eagle.username | root | String | Kafka Eagle store database username. |
kafka.eagle.password | 123456 | String | Kafka Eagle store database password. |
SQLite
Name | Default | Type | Describe |
---|
kafka.eagle.driver | org.sqlite.JDBC | String | Kafka Eagle store database driver. |
kafka.eagle.url | jdbc:sqlite:/hadoop/kafka-eagle/db/ke.db | String | Kafka Eagle store database url. |
kafka.eagle.username | root | String | Kafka Eagle store database username. |
kafka.eagle.password | 123456 | String | Kafka Eagle store database password. |
2.5 启动
cd ${KE_HOME}/bin
chmod +x ke.sh
./ke.sh start
3 访问
页面访问地址:http://host:port/ke
4 附录:
官方文档:https://ke.smartloli.org/