应用场景: kafka的多消费者消费同一个topic的数据,并且保证每条记录都能被消费掉
实现方式: topic要创建多个partition(partition的个数要大于等于消费者的个数),多个消费者在同一个组之内
第一步: 创建topic的多partition,并进行验证,如下所示:
[root@hadoop ~]# kafka-topics.sh --list --zookeeper hadoop:2181
[root@hadoop ~]# kafka-topics.sh --create --zookeeper hadoop:2181 --topic kafkatest --partitions 3 --replication-factor 1
Created topic "kafkatest".
[root@hadoop ~]# kafka-topics.sh --list --zookeeper hadoop:2181
kafkatest
[root@hadoop ~]# kafka-topics.sh --describe --zookeeper hadoop:2181
Topic:kafkatest PartitionCount:3 ReplicationFactor:1 Configs:
Topic: kafkatest Partition: 0 Leader: 0 Replicas: 0 Isr: 0
Topic: kafkatest Partition: 1 Leader: 0 Replicas: 0 Isr: 0
Topic: kafkatest Partition: 2 Leader: 0 Replicas: 0 Isr: 0
结果显示,已经topic进行创建成功,并且partition的个数为3 (因为后面验证的kafka的消费者的个数为2)
第二步: 增加kafka的消费者配置文件consumer2.properties,为后面的测试做准备,如下所示: