Kafka集群部署

IP	主机名	角色	brokerId
192.168.1.130	test1	broker server	0
192.168.1.140	test2	broker server	1
192.168.1.150	test3	broker server	2

kafka依赖zookeeper环境

部署文档 https://blog.csdn.net/weixin_43497190/article/details/121570618

下载安装包

    官网地址

• Apache ZooKeeperhttps://zookeeper.apache.org/releases.html#download

解压并重命名

[root@test1 ~]# tar zxf kafka_2.12-2.5.1.tgz -C /data/
[root@test1 ~]# mv /data/kafka_2.12-2.5.1/ /data/kafka

修改配置文件

[root@test1 ~]# cat /data/kafka/config/server.properties 
# 指定节点borkerid，集群id要唯一
broker.id=0
# 指定监听的地址及端口号，内网ip
listeners=PLAINTEXT://192.168.1.130:9092
# 如果需要开放外网访问，则在该配置项指定外网ip
advertised.listeners=PLAINTEXT://192.168.1.130:9092
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
# 指定kafka日志文件的存储目录
log.dirs=/data/kafka_data/kafka-logs  #此目录需要创建
num.partitions=1
num.recovery.threads.per.data.dir=1
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
# 指定zookeeper的连接地址
zookeeper.connect=192.168.1.130:2181,192.168.1.140:2181,192.168.1.150:2181
zookeeper.connection.timeout.ms=18000
group.initial.rebalance.delay.ms=0

启动kafka &后台运行

[root@test1 ~]# /data/kafka/bin/kafka-server-start.sh /data/kafka/config/server.properties &

查看端口号

[root@test1 ~]# netstat -anpt | grep 9092
tcp6       0      0 192.168.1.130:9092      :::*                    LISTEN      2700/java 

test2和test3修改以下配置

#test2
broker.id=1
listeners=PLAINTEXT://192.168.1.140:9092
advertised.listeners=PLAINTEXT://192.168.1.140:9092
log.dirs=/data/kafka_data/kafka-logs  
#test3
broker.id=1
listeners=PLAINTEXT://192.168.1.150:9092
advertised.listeners=PLAINTEXT://192.168.1.150:9092
log.dirs=/data/kafka_data/kafka-logs  

配置完成启动test2和test3

启动成功后运行Zookeeper客户端，在/brokers/ids下有相应的brokerId数据代表集群搭建成功

[root@test1 ~]# /data/zookeeper/bin/zkCli.sh
[zk: localhost:2181(CONNECTED) 0] ls /brokers/ids
[0, 1, 2]

kafka集群测试

任意一台创建主题
[root@test1 bin]# ./kafka-topics.sh --create --zookeeper 192.168.1.130:2181,192.168.1.140:2181,192.168.1.150:2181 --topic test-kafka --partitions 3 --replication-factor 1
Created topic test-kafka.
任意一台向主题中写入内容
[root@test2 bin]# ./kafka-console-producer.sh --broker-list 192.168.1.130:9092,192.168.1.140:9092,192.168.1.150:9092 --topic test-kafka
>hello world	
 ctrl+d 退出

查看消费信息
[root@test2 bin]# ./kafka-console-consumer.sh --bootstrap-server 192.168.1.130:9092 --topic test-kafka --from-beginning
hello world
[root@test2 bin]# ./kafka-console-consumer.sh --bootstrap-server 192.168.1.140:9092 --topic test-kafka --from-beginning
hello world
[root@test2 bin]# ./kafka-console-consumer.sh --bootstrap-server 192.168.1.150:9092 --topic test-kafka --from-beginning
hello world

集群测试成功