Kafka容错性测试

程研板

已于 2022-03-12 21:23:38 修改

阅读量2k

点赞数

分类专栏： # Kafka 文章标签： kafka zookeeper hadoop

于 2022-03-12 18:46:44 首次发布

本文链接：https://blog.csdn.net/qq_38258720/article/details/123447963

版权

Kafka 专栏收录该内容

10 篇文章 1 订阅

订阅专栏

本文详细介绍了如何创建并配置3个Kafka broker，展示在kill掉部分节点后，系统的容错性和恢复过程。重点在于通过实例演示了topic管理、消费者和生产者的操作，以及集群在不同故障情况下的行为。

摘要由CSDN通过智能技术生成

一.创建3个配置文件

server.properties：

broker.id=0
listeners=PLAINTEXT://hadoop000:9092
log.dirs=/home/hadoop/app/tmp/kafka-log

server-1.properties：

broker.id=1
listeners=PLAINTEXT://hadoop000:9093
log.dirs=/home/hadoop/app/tmp/kafka-log-1

server-2.properties：

broker.id=2
listeners=PLAINTEXT://hadoop000:9094
log.dirs=/home/hadoop/app/tmp/kafka-log-2

二.启动zk和kafka

[hadoop@hadoop000 bin]$ ./zkServer.sh start
[hadoop@hadoop000 config]$ kafka-server-start.sh -daemon $KAFKA_HOME/config/server.properties
[hadoop@hadoop000 config]$ kafka-server-start.sh -daemon $KAFKA_HOME/config/server-1.properties
[hadoop@hadoop000 config]$ kafka-server-start.sh -daemon $KAFKA_HOME/config/server-2.properties

[hadoop@hadoop000 config]$ jps -m
10304 QuorumPeerMain /home/hadoop/app/zookeeper-3.4.5-cdh5.15.1/bin/../conf/zoo.cfg
10611 Kafka /home/hadoop/app/kafka_2.11-2.2.1/config/server.properties
10967 Kafka /home/hadoop/app/kafka_2.11-2.2.1/config/server-1.properties
11333 Kafka /home/hadoop/app/kafka_2.11-2.2.1/config/server-2.properties
12837 ConsoleConsumer --bootstrap-server hadoop000:9092,hadoop000:9093,hadoop000:9094 --topic access-topic-prod
13401 ConsoleProducer --broker-list hadoop000:9092,hadoop000:9093,hadoop000:9094 --topic access-topic-prod
13966 Jps -m

三.创建topic、消费者、生产者

[hadoop@hadoop000 config]$ kafka-topics.sh --create --bootstrap-server hadoop000:9092,hadoop000:9093,hadoop000:9094 --replication-factor 3 --partitions 1 --topic access-topic-prod

[hadoop@hadoop000 bin]$ kafka-console-consumer.sh --bootstrap-server hadoop000:9092,hadoop000:9093,hadoop000:9094 --topic access-topic-prod

[hadoop@hadoop000 ~]$ kafka-console-producer.sh --broker-list hadoop000:9092,hadoop000:9093,hadoop000:9094 --topic access-topic-prod

四.三个broker

[hadoop@hadoop000 config]$ kafka-topics.sh --describe --bootstrap-server hadoop000:9092,hadoop000:9093,hadoop000:9094 --topic access-topic-prod
Topic:access-topic-prod	PartitionCount:1	ReplicationFactor:3	Configs:segment.bytes=1073741824
	Topic: access-topic-prod	Partition: 0	Leader: 2	Replicas: 2,1,0	Isr: 2,1,0

五.kill掉一个，依然能工作

[hadoop@hadoop000 config]$ kill -9 11333

[hadoop@hadoop000 config]$ jps -m
10304 QuorumPeerMain /home/hadoop/app/zookeeper-3.4.5-cdh5.15.1/bin/../conf/zoo.cfg
10611 Kafka /home/hadoop/app/kafka_2.11-2.2.1/config/server.properties
10967 Kafka /home/hadoop/app/kafka_2.11-2.2.1/config/server-1.properties
12837 ConsoleConsumer --bootstrap-server hadoop000:9092,hadoop000:9093,hadoop000:9094 --topic access-topic-prod
13401 ConsoleProducer --broker-list hadoop000:9092,hadoop000:9093,hadoop000:9094 --topic access-topic-prod
13979 Jps -m

[hadoop@hadoop000 config]$ kafka-topics.sh --describe --bootstrap-server hadoop000:9092,hadoop000:9093,hadoop000:9094 --topic access-topic-prod
[2022-03-12 18:17:11,192] WARN [AdminClient clientId=adminclient-1] Connection to node -3 (hadoop000/192.168.198.128:9094) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
Topic:access-topic-prod	PartitionCount:1	ReplicationFactor:3	Configs:segment.bytes=1073741824
	Topic: access-topic-prod	Partition: 0	Leader: 1	Replicas: 2,1,0	Isr: 1,0

六.kill掉两个，依然能工作

[hadoop@hadoop000 config]$ kill -9 10967

[hadoop@hadoop000 config]$ jps -m
10304 QuorumPeerMain /home/hadoop/app/zookeeper-3.4.5-cdh5.15.1/bin/../conf/zoo.cfg
10611 Kafka /home/hadoop/app/kafka_2.11-2.2.1/config/server.properties
12837 ConsoleConsumer --bootstrap-server hadoop000:9092,hadoop000:9093,hadoop000:9094 --topic access-topic-prod
13401 ConsoleProducer --broker-list hadoop000:9092,hadoop000:9093,hadoop000:9094 --topic access-topic-prod
15148 Jps -m

[hadoop@hadoop000 config]$ kafka-topics.sh --describe --bootstrap-server hadoop000:9092,hadoop000:9093,hadoop000:9094 --topic access-topic-prod
[2022-03-12 18:18:38,990] WARN [AdminClient clientId=adminclient-1] Connection to node -3 (hadoop000/192.168.198.128:9094) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
Topic:access-topic-prod	PartitionCount:1	ReplicationFactor:3	Configs:segment.bytes=1073741824
	Topic: access-topic-prod	Partition: 0	Leader: 0	Replicas: 2,1,0	Isr: 0

七.重启一个，Isr会恢复

[hadoop@hadoop000 bin]$ kafka-topics.sh --describe --bootstrap-server hadoop000:9092,hadoop000:9093,hadoop000:9094 --topic access-topic-prod
Topic:access-topic-prod	PartitionCount:1	ReplicationFactor:3	Configs:segment.bytes=1073741824
	Topic: access-topic-prod	Partition: 0	Leader: 1	Replicas: 2,1,0	Isr: 1,0
	
[hadoop@hadoop000 bin]$ kafka-server-start.sh -daemon $KAFKA_HOME/config/server-2.properties

[hadoop@hadoop000 bin]$ kafka-topics.sh --describe --bootstrap-server hadoop000:9092,hadoop000:9093,hadoop000:9094 --topic access-topic-prod
Topic:access-topic-prod	PartitionCount:1	ReplicationFactor:3	Configs:segment.bytes=1073741824
	Topic: access-topic-prod	Partition: 0	Leader: 1	Replicas: 2,1,0	Isr: 1,0,2