kafka集群搭建

1、环境介绍
操作系统：centos 7.9
jdk版本：8u291
kafka版本：2.8.0
kafka下载地址：
https://kafka.apache.org/downloads

节点清单：
10.99.27.101 kafka01.wtown.com 4核心 8G内存 500G硬盘
10.99.27.102 kafka02.wtown.com 4核心 8G内存 500G硬盘
10.99.27.103 kafka03.wtown.com 4核心 8G内存 500G硬盘

2、设置主机名及host文件（三台机器）

10.99.27.101	kafka01.wtown.com
10.99.27.102	kafka02.wtown.com
10.99.27.103	kafka03.wtown.com

3、关闭防火墙和selinux（三台机器）

4、创建数据目录/data，并挂载数据盘（三台机器）

mkdir /data

https://blog.csdn.net/zyj81092211/article/details/118054000

5、配置jdk
https://blog.csdn.net/zyj81092211/article/details/118055068

6、创建zookeeper集群
https://blog.csdn.net/zyj81092211/article/details/118066724

7、上传软件到服务器解压并重命名为kafka（三台机器）

8、创建软连接到 /usr/local下（三台机器）

ln -s /data/kafka /usr/local/kafka

9、更改配置文件（三台机器）

编辑配置文件server.properties，替换文件内容为下

vi /data/kafka/config/server.properties

kafka01.wtown.com：

broker.id=1
listeners=PLAINTEXT://kafka01.wtown.com:9092
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/data/kafka/logs
num.partitions=1
num.recovery.threads.per.data.dir=1
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
zookeeper.connect=zk01.wtown.com:2181,zk02.wtown.com:2181,zk03.wtown.com:2181/kafka
zookeeper.connection.timeout.ms=18000
group.initial.rebalance.delay.ms=0

kafka02.wtown.com:

broker.id=2
listeners=PLAINTEXT://kafka02.wtown.com:9092
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/data/kafka/logs
num.partitions=1
num.recovery.threads.per.data.dir=1
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
zookeeper.connect=zk01.wtown.com:2181,zk02.wtown.com:2181,zk03.wtown.com:2181/kafka
zookeeper.connection.timeout.ms=18000
group.initial.rebalance.delay.ms=0

kafka03.wtown.com:

broker.id=3
listeners=PLAINTEXT://kafka03.wtown.com:9092
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/data/kafka/logs
num.partitions=1
num.recovery.threads.per.data.dir=1
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
zookeeper.connect=zk01.wtown.com:2181,zk02.wtown.com:2181,zk03.wtown.com:2181/kafka
zookeeper.connection.timeout.ms=18000
group.initial.rebalance.delay.ms=0

10、添加环境变量（三台机器）

export KAFKA_HOME=/data/kafka
export PATH=$PATH:$KAFKA_HOME/bin

11、启动集群（三台机器）

kafka-server-start.sh -daemon /data/kafka/config/server.properties

12、查看状态

13、测试

创建topic

kafka-topics.sh --create --partitions 3 --replication-factor 2 --topic kafka-test --zookeeper zk01.wtown.com:2181,zk02.wtown.com:2181,zk03.wtown.com:2181/kafka

查看topic

kafka-topics.sh --list --zookeeper zk01.wtown.com:2181,zk02.wtown.com:2181,zk03.wtown.com:2181/kafka

模拟生产者

kafka-console-producer.sh --broker-list kafka01.wtown.com:9092,kafka02.wtown.com:9092,kafka03.wtown.com:9092 --topic kafka-test

模拟消费者

kafka-console-consumer.sh --bootstrap-server kafka01.wtown.com:9092,kafka02.wtown.com:9092,kafka03.wtown.com:9092 --topic kafka-test

查看指定topic

kafka-topics.sh --describe --zookeeper zk01.wtown.com:2181,zk02.wtown.com:2181,zk03.wtown.com:2181/kafka --topic kafka-test

删除topic

kafka-topics.sh --delete --zookeeper zk01.wtown.com:2181,zk02.wtown.com:2181,zk03.wtown.com:2181/kafka --topic kafka-test

老版本还需进入zookeeper中删除
进入zookeeper中删除topic信息，完成删除

zkCli.sh 

deleteall /kafka/brokers/topics/kafka-test