一、安装环境:kafka_2.12-2.3.0(Scala 2.12,kafka 2.3.0)、zookeeper-3.4.14、centos7、3节点IP:192.168.56.66,192.168.56.80,192.168.56.88。
二、官网安装:http://kafka.apache.org/quickstart
官网最新正式版下载:http://kafka.apache.org/downloads,选择kafka_2.12-2.3.0.tgz
三、下载好后winscp传到centos66节点,解压、改名、创建日志文件夹:
tar -xzf kafka_2.12-2.3.0.tgz
mv kafka_2.12-2.3.0 kafka2.3.0
cd kafka2.3.0/
mkdir logs
ls
bin config libs LICENSE logs NOTICE site-docs
配置文件修改:
cd config/
pwd
/root/kafka2.3.0/config
vi server.properties
默认值(broker.id:broker的标识,listeners:kafka-server监听的主机与端口,log.dirs:日志存放路径,zookeeper.connect:Zookeeper集群地址):
broker.id=0
#listeners=PLAINTEXT://:9092
log.dirs=/tmp/kafka-logs
zookeeper.connect=localhost:2181
修改后的值:
broker.id=0
listeners=PLAINTEXT://centos66:9092
log.dirs=/root/kafka2.3.0/logs
zookeeper.connect=192.168.56.66:2181,192.168.56.80:2181,192.168.56.88:2181
修改后整个文件夹scp到其它2节点,继续修改broker.id、listeners值:
cd /root
scp -r kafka2.3.0 root@centos80:/root/
scp -r kafka2.3.0 root@centos88:/root/
ssh centos80
cd kafka2.3.0/config/
vi server.properties
broker.id=1
listeners=PLAINTEXT://centos80:9092
ssh centos88
cd kafka2.3.0/config/
vi server.properties
broker.id=2
listeners=PLAINTEXT://centos88:9092
启动kafka:
启动kafka前,3节点间的ssh免密通信参考:https://blog.csdn.net/yzh_1346983557/article/details/100080838
[root@centos66 ~]# cat /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.56.66 centos66
192.168.56.80 centos80
192.168.56.88 centos88
启动kafka前,启动zookeeper集群参考:https://blog.csdn.net/yzh_1346983557/article/details/102705087
[root@centos66 bin]# jps
29797 QuorumPeerMain
24699 Jps
最后去3个节点分别启动kafka-server:
[root@centos66 ~]# cd /root/kafka2.3.0/
[root@centos66 kafka2.3.0]# bin/kafka-server-start.sh -daemon config/server.properties
[root@centos66 kafka2.3.0]# jps
25537 Kafka
26100 Jps
21035 QuorumPeerMain
[root@centos66 kafka2.3.0]# ssh centos80
[root@centos80 ~]# cd /root/kafka2.3.0/
[root@centos80 kafka2.3.0]# bin/kafka-server-start.sh -daemon config/server.properties
[root@centos66 kafka2.3.0]# ssh centos88
[root@centos88 ~]# cd /root/kafka2.3.0/
[root@centos88 kafka2.3.0]# bin/kafka-server-start.sh -daemon config/server.properties
四、kafka集群的验证使用
创建3副本、2分区的topic:yzh
[root@centos66 ~]# cd /root/kafka2.3.0/
[root@centos66 kafka2.3.0]# bin/kafka-topics.sh --create --bootstrap-server centos66:9092 --replication-factor 3 --partitions 2 --topic yzh
[root@centos66 kafka2.3.0]# bin/kafka-topics.sh --describe --bootstrap-server centos66:9092 --topic yzh
Topic:yzh PartitionCount:2 ReplicationFactor:3 Configs:segment.bytes=1073741824
Topic: yzh Partition: 0 Leader: 1 Replicas: 1,0,2 Isr: 1,0,2
Topic: yzh Partition: 1 Leader: 0 Replicas: 0,2,1 Isr: 0,2,1
第一行给出了topic的摘要,下面每一行给出了每个分区的信息。
- “Leader”是负责分区的所有读取和写入的节点。每个节点将会被随机选为partition的Leader(领导者)。
- “Replicas”是为分区复制日志的节点列表,不管它们是引导者还是当前是否处于活动状态。
- “ isr”是“同步”副本的集合。这是Replicas列表的子集,当前仍处于活动状态并追随领导者。
生产数据:
[root@centos66 kafka2.3.0]# bin/kafka-console-producer.sh --broker-list centos66:9092 --topic yzh
>my test message 1
>my test message 2
>^C
消费数据:
[root@centos66 kafka2.3.0]# bin/kafka-console-consumer.sh --bootstrap-server centos66:9092 --from-beginning --topic yzh
my test message 1
my test message 2
^C
kafka集群HA的验证:由于Partition: 0 Leader: 1 且 Partition: 1 Leader: 0,分区0的Leader是1以及分区1的Leader是0,所以kill 0和1的kafka-server,观察broker.id=2的kafka-server能否继续正常提供topic yzh的数据。
[root@centos66 kafka2.3.0]# jps
29797 QuorumPeerMain
29751 Kafka
926 Jps
[root@centos66 kafka2.3.0]# kill -9 29751
[root@centos66 kafka2.3.0]# jps
29797 QuorumPeerMain
8650 Jps
[root@centos66 kafka2.3.0]# ssh centos80
[root@centos80 ~]# jps
7392 Jps
25537 Kafka
21035 QuorumPeerMain
[root@centos80 ~]# kill -9 25537
[root@centos80 ~]# jps
9797 Jps
21035 QuorumPeerMain
[root@centos66 kafka2.3.0]# bin/kafka-topics.sh --describe --bootstrap-server centos88:9092 --topic yzh
Topic:yzh PartitionCount:2 ReplicationFactor:3 Configs:segment.bytes=1073741824
Topic: yzh Partition: 0 Leader: 2 Replicas: 1,0,2 Isr: 2
Topic: yzh Partition: 1 Leader: 2 Replicas: 0,2,1 Isr: 2
[root@centos66 kafka2.3.0]# bin/kafka-console-consumer.sh --bootstrap-server centos88:9092 --from-beginning --topic yzh
my test message 1
my test message 2
^C
发现broker.id=2的kafka-server依然可以正常提供topic yzh的数据,说明kafka集群HA,符合高可用。
再次启动centos66节点的broker.id=0的kafka-server和centos80节点的broker.id=1的kafka-server,查看topic yzh的状态:
[root@centos80 kafka2.3.0]# bin/kafka-topics.sh --describe --bootstrap-server centos88:9092 --topic yzh
Topic:yzh PartitionCount:2 ReplicationFactor:3 Configs:segment.bytes=1073741824
Topic: yzh Partition: 0 Leader: 2 Replicas: 1,0,2 Isr: 2,0,1
Topic: yzh Partition: 1 Leader: 2 Replicas: 0,2,1 Isr: 2,0,1
附:
消费数据时指定Partition:
[root@centos88 kafka2.3.0]# bin/kafka-console-consumer.sh --bootstrap-server centos66:9092 --from-beginning --topic yzh --partition 1
或者
[root@centos88 kafka2.3.0]# bin/kafka-console-consumer.sh --bootstrap-server centos66:9092 --from-beginning --topic yzh -partition 0
显示集群所有topic:
[root@centos88 kafka2.3.0]# bin/kafka-topics.sh --list --bootstrap-server centos88:9092
Kafka在启动时会在zookeeper中/brokers/ids路径下创建一个与当前broker.id为名称的虚节点,Kafka的健康状态检查就依赖于此节点。当broker下线时,该虚节点会自动删除,其他broker或者客户端通过判断/brokers/ids路径下是否有此broker的id来确定该broker的健康状态:
[root@centos52 bin]# pwd
/usr/local/zookeeper-3.4.14/bin
[root@centos52 bin]# ./zkCli.sh
[zk: localhost:2181(CONNECTED) 1] ls /brokers/ids
[0, 1, 2]
[zk: localhost:2181(CONNECTED) 7] get /brokers/ids/0
{"listener_security_protocol_map":{"PLAINTEXT":"PLAINTEXT"},"endpoints":["PLAINTEXT://centos52:9092"],"jmx_port":-1,"host":"centos52","timestamp":"1571897362384","port":9092,"version":4}
cZxid = 0x100000018
ctime = Thu Oct 24 14:09:22 CST 2019
mZxid = 0x100000018
mtime = Thu Oct 24 14:09:22 CST 2019
pZxid = 0x100000018
cversion = 0
dataVersion = 1
aclVersion = 0
ephemeralOwner = 0x1000117e12b0000
dataLength = 186
numChildren = 0