Kafka-Eagle 2.0.5安装

Kafka-Eagle 2.0.5安装

Kafka Eagle是一款由国内公司开源的Kafka集群监控系统，可以用来监视kafka集群的broker状态、Topic信息、IO、内存、consumer线程、偏移量等信息，并进行可视化图表展示。独特的KQL还可以通过SQL在线查询kafka中的数据。

1. 安装准备

zookeeper-3.4.5已安装

kafka_2.11-2.2.1已安装

kafka-eagle-web-2.0.5-bin.tar.gz

kafka-eagle官网http://www.kafka-eagle.org/ 目前只能下载最新版本kafka-eagle-web-3.0.1

1. 安装配置

2.1 数据库安装

以root用户登陆拷贝mysql5.7.19介质到mysql 服务器

tar -zxvf mysql-5.7.19-1.el7.x86_64.rpm-bundle.tar -C mysql

卸载系统自带mariadb-lib：

yum list installed | grep mariadb rpm -e --nodeps mariadb-libs.x86_64

安装mysql

rpm -ivh mysql-community-common-5.7.19-1.el7.x86_64.rpm rpm -ivh mysql-community-libs-5.7.19-1.el7.x86_64.rpm rpm -ivh mysql-community-libs-compat-5.7.19-1.el7.x86_64.rpm rpm -ivh mysql-community-client-5.7.19-1.el7.x86_64.rpm rpm -ivh mysql-community-server-5.7.19-1.el7.x86_64.rpm

检查服务自启动状态

systemctl is-enabled mysqld

启动mysql：

systemctl start mysqld

2.2 修改密码

默认mysql 5.7.19安装后，初始root用户临时密码需要在启动mysql服务后的/var/log/mysqld.log日志中查看：

cat /var/log/mysqld.log | grep password

修改root密码为“[Root123]”：

passwd=$(cat /var/log/mysqld.log  | grep -i password | head -n 1 | awk '{print $11}')

echo $passwd

mysqladmin -uroot -p"$passwd" password '[Root123]'

初始化数据库

/usr/bin/mysql_secure_installation

Securing the MySQL server deployment. Enter password for user root: 输入前面修改的root密码：[Root123] The 'validate_password' plugin is installed on the server. The subsequent steps will run with the existing configuration of the plugin. Using existing password for root. Estimated strength of the password: 100 Change the password for root ? ((Press y|Y for Yes, any other key for No) : Enter  ... skipping. By default, a MySQL installation has an anonymous user, allowing anyone to log into MySQL without having to have a user account created for them. This is intended only for testing, and to make the installation go a bit smoother. You should remove them before moving into a production environment. Remove anonymous users? (Press y|Y for Yes, any other key for No) : y Success. Normally, root should only be allowed to connect from 'localhost'. This ensures that someone cannot guess at the root password from the network. Disallow root login remotely? (Press y|Y for Yes, any other key for No) : Enter  ... skipping. By default, MySQL comes with a database named 'test' that anyone can access. This is also intended only for testing, and should be removed before moving into a production environment. Remove test database and access to it? (Press y|Y for Yes, any other key for No) : y  - Dropping test database... Success.  - Removing privileges on test database... Success. Reloading the privilege tables will ensure that all changes made so far will take effect immediately. Reload privilege tables now? (Press y|Y for Yes, any other key for No) : y Success. All done!

增加mysql root用户远程登陆配置

mysql -uroot -p'[Root123]' mysql -e "GRANT ALL PRIVILEGES ON *.* TO 'root'@'%' IDENTIFIED BY '[Root123]' WITH GRANT OPTION;"

mysql -uroot -p'[Root123]' mysql -e "FLUSH PRIVILEGES;"

mysql -uroot -p'[Root123]' mysql -e "select user,host,authentication_string from user;"

2.3 创建数据库

mysql -uroot -p[Root123]

create database kafkaefak DEFAULT CHARACTER SET utf8;

2.4 解压安装

1、tar -zxvf kafka-eagle-web-2.0.5-bin.tar.g

2、赋予ke.sh执行权限 cd /data/kafka-eagle-web-2.0.5/bin/ chmod 777 ke.sh

2.5 配置文件修改

vim /data/kafka-eagle-web-2.0.5/conf/system-config.properties

# multi zookeeper & kafka cluster list ###################################### kafka.eagle.zk.cluster.alias=cluster1 cluster1.zk.list=0.0.5.92:2181,0.0.5.93:2181,0.0.5.4:2181

# kafka eagle webui port ###################################### kafka.eagle.webui.port=8048

# delete kafka topic token ###################################### kafka.eagle.topic.token=keadmin

###################################### # kafka mysql jdbc driver address ###################################### kafka.eagle.driver=com.mysql.cj.jdbc.Driver kafka.eagle.url=jdbc:mysql://0.0.5.92:3306/kafkaefak?useUnicode=true&characterEncoding=UTF-8&zeroDateTimeBehavior=convertToNull kafka.eagle.username=root kafka.eagle.password=[Root123]

Kafka集群的各个Broker节点JMX_PORT开启（这个Kafka默认是不开启）

cd /opt/kafka/kafka_2.11-2.2.1/bin/ vim kafka-server-start.sh

if [ "x$KAFKA_HEAP_OPTS" = "x" ]; then  export KAFKA_HEAP_OPTS="-server -Xms2G -Xmx2G -XX:PermSize=128m -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -XX:ParallelGCThreads=8 -XX:ConcGCThreads=5 -XX:InitiatingHeapOccupancyPercent=70"  export JMX_PORT="9999" if

有时候可能会在日志中发现一些连接超时或是空指针异常，对于这类问题，首先需要检测Kafka集群的各个Broker节点JMX_PORT是否开启

2.6 配置环境变量

export KE_HOME=/data/kafka-eagle-web-2.0.5 export PATH=$PATH:$KE_HOME/bin

source /etc/profile

2.7启动

cd /data/kafka-eagle-web-2.0.5/

bin/ke.sh start

1. 安装页面

3.1 地址

http://0.0.5.92:8048

账号：admin 密码：123456

3.2 介绍

这里包含了kafka集群的节点、topic、zk信息、消费者组、Topic Top10的日志大小、Topic Top10的容量等等一些信息。

Kafka-eagle展示大屏：从左到右展示的是过去7天的生产记录、从左到右展示的是过去7天的消费记录、总的topic记录数、今天生产者的信息、今天消费者的信息、今天滞后的信息

该模块包含监控Kafka集群和Zookeeper集群的核心指标，包含Kafka的消息发送趋势、消息大小接收与发送趋势、Zookeeper的连接数趋势等。同时，还支持查看Broker的瞬时指标数据。

另外需要注意的是，配置钉钉报警时，关键字一定要是Kafka Eagle Alarm xxx Notice几个单词中的一个单词或字母；配置邮件报警的话，要自己封装一个接口