kafka 2.8.0 without zookerper 部署

kafka 2.8.0 without zookerper 部署

1.前言

Kafka 2.8.0 可以不需要zookerper，没有zk的参与将会使kafka的吞吐量得到更大的提高，具体可看官网，这篇文章主要是针对kafka 2.8.0 kafka KRaft mode 的安装部署，以三台机器为例，由于网上资料较少，大部分都是靠自己摸索，若文中有理解不对的地方，可随时赐教，共同探讨。

2.下载安装包 (https://kafka.apache.org/downloads)

3.解压安装包

$ tar -zxvf kafka_2.12-2.8.0.tgz -C /data1/opt/module

4.修改server.properties文件

$ cd /data1/opt/module/kafka_2.12-2.8.0/config/kraft
$ vim server.properties

注意：

1. 设置node.id,集群中node.id不可重复
2. 设置controller.quorum.voters，所有controller需要全部配置
3. log.dirs可配置多目录
4. 配置文件中所有localhost,更改为ip或主机名

5.创建cluster ID

在任意一台机器执行

$ ./bin/kafka-storage.sh random-uuid

6.同步cluster ID

在所有节点执行

$ ./bin/kafka-storage.sh format -t <uuid> -c ./config/kraft/server.properties

7.启动kafka

在每台节点上执行，启动kafka

$ ./bin/kafka-server-start.sh -daemon /data1/opt/module/kafka_2.12-2.8.0/config/kraft/server.properties

8.指令测试

(1) 查看当前服务器中的所有topic

$ ./bin/kafka-topics.sh --bootstrap-server hadoop101,hadoop102,hadoop103 --list
__consumer_offsets
four
one
three
two

(2) 创建topic

$ ./bin/kafka-topics.sh --bootstrap-server hadoop101,hadoop102,hadoop103 --create --replication-factor 2 --partitions 2 --topic five
Created topic five.

(3) 删除topic

$ ./bin/kafka-topics.sh --bootstrap-server hadoop101 --delete --topic one

(4) 发送消息

$ ./bin/kafka-console-producer.sh --broker-list hadoop101,hadoop102,hadoop103 --topic three
>one
>two
>three
>

(5) 消费消息

$ ./bin/kafka-console-consumer.sh --bootstrap-server hadoop101,hadoop102,hadoop103 --topic three –group g1

$ ./bin/kafka-console-consumer.sh --bootstrap-server hadoop101 --from-beginning --topic three
one
three
saas
two
five

(6) 查看某个Topic的详情

$ ./bin/kafka-topics.sh --bootstrap-server hadoop101 --describe --topic three
Topic: three TopicId: 6enhWA-nRh6bMPmk6jWw0w PartitionCount: 2 ReplicationFactor: 2 Configs: segment.bytes=1073741824
Topic: three Partition: 0 Leader: 1 Replicas: 1,3 Isr: 1,3
Topic: three Partition: 1 Leader: 3 Replicas: 3,1 Isr: 3,1

(7) 修改分区数(KRaft mode 不支持)

$ ./bin/kafka-topics.sh --bootstrap-server hadoop101 --alter --topic two --partitions 2
Error while executing topic command : The broker does not support CREATE_PARTITIONS
[2021-07-20 15:07:36,742] ERROR org.apache.kafka.common.errors.UnsupportedVersionException: The broker does not support CREATE_PARTITIONS
(kafka.admin.TopicCommand$)

9.压测

(1) Kafka Producer压力测试

$ ./bin/kafka-producer-perf-test.sh --topic test --record-size 100 --num-records 100000 --throughput -1 --producer-props bootstrap.servers=hadoop101,hadoop102,hadoop103
100000 records sent, 49455.984174 records/sec (4.72 MB/sec), 561.53 ms avg latency, 878.00 ms max latency, 595 ms 50th, 839 ms 95th, 871 ms 99th, 877 ms 99.9th.

(2) Kafka Consumer压力测试

$ ./bin/kafka-consumer-perf-test.sh --broker-list hadoop101,hadoop102,hadoop103 --topic test --fetch-size 10000 --messages 10000000 --threads 1
2021-07-20 15:20:42:518, 2021-07-20 15:20:57:845, 9.5367, 0.6222, 100000, 6524.4340, 3556, 11771, 0.8102, 8495.4549