首发地址:https://www.pengtao.vip/archives/bu-shu-kafkahe-zookeeperji-qun–jun-san-jie-dian-
1、系统环境
本次部署为三台服务器部署zookeeper和Kafka集群。
操作系统 | zookeeper版本 | Kafka版本 |
---|---|---|
CentOS Linux release 7.8.2003 | zookeeper-3.7.0 | kafka_2.13-3.0.0 |
自行安装 JDK,启动zookeeper需要Java环境
2、配置Zookeeper集群(三节点一样)
1、创建Zk server
的集群安装目录
[root@rac2 /]# mkdir /data
[root@rac2 /]# mkdir -p /data/zookeeper
2、上传zookeeper软件并解压到相应目录
[root@rac2 data]# tar -zxvf apache-zookeeper-3.7.0-bin.tar.gz -C /data/zookeeper/
3、创建日志目录
#创建快照日志存放目录:
mkdir -p /data/zookeeper/dataDir
#创建事务日志存放目录:
mkdir -p /data/zookeeper/dataLogDir
4、修改zookeeper配置文件
[root@rac2 data]# cd /data/zookeeper/conf/
[root@rac2 conf]# mv zoo_sample.cfg zoo.cfg
[root@rac2 conf]# vim zoo.cfg
#配置内容如下,其他两个节点一样
[root@rac2 conf]# cat zoo.cfg |grep -v ^#|grep -v ^$
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/data/zookeeper/dataDir
dataLogDir=/data/zookeeper/dataLogDir
server.1=192.168.23.161:2888:3888
server.2=192.168.23.162:2888:3888
server.3=192.168.23.163:2888:3888
clientPort=2181
maxClientCnxns=60
在我们配置的dataDir
指定的目录下面,创建一个myid
文件,里面内容为一个数字,用来标识当前主机,对应zoo.cfg文件中配置的server.X
中X
的数字。
[root@rac1 conf]# echo "1" > /data/zookeeper/dataDir/myid #主机1
[root@rac2 conf]# echo "2" > /data/zookeeper/dataDir/myid #主机2
[root@rac3 conf]# echo "3" > /data/zookeeper/dataDir/myid #主机3
5、关闭防火墙
systemctl stop firewalld
systemctl disable firewalld
6、启动zookeeper集群,每个节点都执行
/data/zookeeper/bin/zkServer.sh start
7、查看Zookeeper集群状态
[root@rac1 data]# zookeeper/bin/zkServer.sh status
ZooKeeper JMX enabled by default
Using config: /data/zookeeper/bin/../conf/zoo.cfg
Client port found: 2181. Client address: localhost. Client SSL: false.
Mode: follower
[root@rac2 data]# zookeeper/bin/zkServer.sh status
ZooKeeper JMX enabled by default
Using config: /data/zookeeper/bin/../conf/zoo.cfg
Client port found: 2181. Client address: localhost. Client SSL: false.
Mode: follower
[root@rac3 data]# zookeeper/bin/zkServer.sh status
ZooKeeper JMX enabled by default
Using config: /data/zookeeper/bin/../conf/zoo.cfg
Client port found: 2181. Client address: localhost. Client SSL: false.
Mode: leader
3、配置Kafka集群(三节点一样)
1、 下载Kafka
下载Kafka安装文件到data下
wget https://downloads.apache.org/kafka/3.0.0/kafka_2.13-3.0.0.tgz
2、解压
tar zxf kafka_2.13-3.0.0.tgz
3、修改配置文件
[root@rac1 kafka_2.13-3.0.0]# cat config/server.properties |grep -v ^#|grep -v ^$
broker.id=0 #另外两个修改为1,2
listeners=PLAINTEXT://192.168.23.161:9092 #对应本机ip
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/tmp/kafka-logs
num.partitions=1
num.recovery.threads.per.data.dir=1
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
zookeeper.connect=192.168.23.161:2181,192.168.23.162:2181,192.168.23.163:2181
delete.topic.enble=true
zookeeper.connection.timeout.ms=18000
group.initial.rebalance.delay.ms=0
4、启动kafka(在保证zookeeper集群没问题的前提下启动)
[root@rac2 kafka_2.13-3.0.0]# nohup bin/kafka-server-start.sh config/server.properties &
5、功能测试
#主机1创建一个topic
[root@rac1 kafka_2.13-3.0.0]# bin/kafka-topics.sh --create --topic producer --bootstrap-server 192.168.23.161:9092,192.168.23.162:9092,192.168.23.163:9092 --partitions 1 --replication-factor 1
#查看所有topic
[root@rac2 kafka_2.13-3.0.0]# bin/kafka-topics.sh --list --bootstrap-server 192.168.23.161:9092
#主机2启动生产者
[root@rac2 kafka_2.13-3.0.0]# bin/kafka-console-producer.sh --broker-list 192.168.23.161:9092,192.168.23.162:9092,192.168.23.163:9092 --topic producer
>999
>777
#主机3启动消费者
[root@rac3 kafka_2.13-3.0.0]# bin/kafka-console-consumer.sh --bootstrap-server 192.168.23.161:9092,192.168.23.162:9092,192.168.23.163:9092 --topic producer --from-beginning
999
777
可以正常生产和消费,部署完成。