Kafka是由Apache软件基金会开发的一个开源流处理平台,由Scala和Java编写。Kafka是一种高吞吐量的分布式发布订阅消息系统,它可以处理消费者在网站中的所有动作流数据。
####一.下载
JDK8,点击进入下载
Kafka2.4.0,下载,新特性MirrorMaker 2.0,[KAFKA-8634] - Update ZooKeeper to 3.5.5
ZooKeeper3.5.5,下载带-bin的apache-zookeeper-3.5.5-bin.tar.gz
####二.安装
2.1 修改主机域名(如果不生效可以参考),设置机器间互信(不是必须,参考)
hostnamectl set-hostname kafka1
cat /etc/hostname
vi /etc/hosts
192.168.0.101 kafka1
192.168.0.102 kafka2
192.168.0.103 kafka3
reboot
2.2安装JDK
将文件拷贝到任意目录,解压;在/usr目录下创建java目录,然后将jdk文件夹移动到java目录
tar -zxvf jdk-8u144-linux-x64.tar.gz
mkdir /usr/java
mv jdk1.8.0_144/ /usr/java
配置环境变量:修改/etc/profile,在末尾添加java信息
export JAVA_HOME=/usr/java/jdk1.8.0_144
export JRE_HOME=${JAVA_HOME}/jre
export CLASSPATH=.:${JAVA_HOME}/lib:${JRE_HOME}/lib
export PATH=${JAVA_HOME}/bin:$PATH
重新加载profile,查看Java安装情况
[root@kafka1 test]# source /etc/profile
[root@kafka1 test]# java -version
java version "1.8.0_144"
Java(TM) SE Runtime Environment (build 1.8.0_144-b01)
Java HotSpot(TM) 64-Bit Server VM (build 25.144-b01, mixed mode)
2.3 安装ZooKeeper(首先安装)
解压zk安装包,修改配置文件
tar -zxvf apache-zookeeper-3.5.5-bin.tar.gz
cd apache-zookeeper-3.5.5-bin/conf/
vi zoo.cfg
添加
dataDir=/data/app/zookeeper/zkdata
server.1=192.168.0.101:2888:3888
server.2=192.168.0.102:2888:3888
server.3=192.168.0.103:2888:3888
设置myid文件,在dataDir目录下,并且要求 myid中的值要与server.X中设置的X相一致, 注意1后面的空格
echo '1' > /data/app/zookeeper/zkdata/myid
启动zk
进去zookeeper目录
启动ZK服务: sh bin/zkServer.sh start
查看ZK服务状态: sh bin/zkServer.sh status
停止ZK服务: sh bin/zkServer.sh stop
重启ZK服务: sh bin/zkServer.sh restart
三台分别配置启动后,查看启动结果(1个leader、2个follower)
[root@kafka1 zookeeper-3.5.5]# sh bin/zkServer.sh status
ZooKeeper JMX enabled by default
Using config: /data/app/zookeeper/zookeeper-3.5.5/bin/../conf/zoo.cfg
Client port found: 2181. Client address: localhost.
Mode: follower
2.4 安装Kafka
解压Kafka安装包,修改配置文件
tar -zxvf kafka_2.12-2.4.0.tgz
vi server.properties
broker.id=1
listeners=PLAINTEXT://192.168.0.101:9092
log.dirs=/data/app/kafka/kafka_log
zookeeper.connect=192.168.0.101:2181,192.168.0.102:2181,192.168.0.103:2181
后台启动Kafka
nohup ./bin/kafka-server-start.sh config/server.properties 1>/dev/null 2>&1 &
查看Kafk集群运行情况
[root@kafka1 kafka_2.12-2.4.0]# jps
30530 Jps
30122 Kafka
29439 QuorumPeerMain(zookeeper)
创建Topic、消费Topic测试
创建topic
./bin/kafka-topics.sh --create --bootstrap-server 192.168.0.101:9092 --replication-factor 1 --partitions 2 --topic test-topic1
命令行生产数据
./bin/kafka-console-producer.sh --broker-list 192.168.0.101:9092 --topic test-topic1
命令行从头消费
./bin/kafka-console-consumer.sh --bootstrap-server 192.168.0.101:9092 --topic test-topic1 --from-beginning
重置指定消费offset(待补齐参数测试)
Reset offsets to a specific offset.
./bin/kafka-consumer-groups.sh --help
./bin/kafka-consumer-groups.sh --to-offset
查看所有topics
./bin/kafka-topics.sh --list --bootstrap-server 192.168.0.101:9092
删除topic
./bin/kafka-topics.sh --delete --bootstrap-server 192.168.0.101:9092 --topic test-topic1
删除所有消费组
./bin/kafka-consumer-groups.sh --bootstrap-server 192.168.0.101:9092 --delete --all-groups
自定义监控Kafka消费进度
举例5秒获取一次消费进度
[root@kafka1 kafka_monitor]# cat kafka_monitor.sh
!/bin/sh
customer=${1}
while [ true ]; do
/bin/sleep 5
date "+%Y-%m-%d %H:%M:%S" >> monitor.log
sh /data/app/kafka/kafka_2.12-2.4.0/bin/kafka-consumer-groups.sh --describe --bootstrap-server 192.168.0.101:9092 --group $customer >> monitor.log
done
返回结果案例:
2020-08-27 13:00:42
GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID
customer-6 A.test-topic11 0 337500 339700 2200 consumer-1-f1474c11-4aeb-402b-b423-df0355f135cb /192.168.0.107 consumer-1
customer-6 A.test-topic11 1 337500 339700 2200 consumer-1-f1474c11-4aeb-402b-b423-df0355f135cb /192.168.0.107 consumer-1
2020-08-27 13:00:46
GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID
customer-6 A.test-topic11 0 407500 409600 2100 consumer-1-f1474c11-4aeb-402b-b423-df0355f135cb /192.168.0.107 consumer-1
customer-6 A.test-topic11 1 407500 409599 2099 consumer-1-f1474c11-4aeb-402b-b423-df0355f135cb /192.168.0.107 consumer-1
2020-08-27 13:00:50
GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID
customer-6 A.test-topic11 0 477500 479488 1988 consumer-1-f1474c11-4aeb-402b-b423-df0355f135cb /192.168.0.107 consumer-1
customer-6 A.test-topic11 1 477500 479450 1950 consumer-1-f1474c11-4aeb-402b-b423-df0355f135cb /192.168.0.107 consumer-1
停止运行Kafka
[root@kafka1 kafka_2.12-2.4.0]#./bin/kafka-server-stop.sh
或
[root@kafka1 kafka_2.12-2.4.0]# kill -9 30122
以上!