kafka 中到底是怎么通过 __consumer_offsets 存储消费位移的

我们知道老版本的消费位移信息是存储的zookeeper 中的， 但是zookeeper 并不适合频繁的写入查询操作，所以在新版本的中消费位移信息存放在了__consumer_offsets内置topic中

那么__consumer_offsets topic 到底是怎么存储我们的消费位移信息呢？

1. 创建topic consumer_offsets_test

bin/kafka-topics.sh --zookeeper localhost:2181 --create --topic consumer_offsets_test --replication-factor 1 --partitions 3

2. 查看topic是否创建成功

bin/kafka-topics.sh --zookeeper localhost:2181 --topic consumer_offsets_test --describe
Topic:consumer_offsets_test    PartitionCount:3    ReplicationFactor:1    Configs:
    Topic: consumer_offsets_test    Partition: 0    Leader: 0    Replicas: 0    Isr: 0
    Topic: consumer_offsets_test    Partition: 1    Leader: 1    Replicas: 1    Isr: 1
    Topic: consumer_offsets_test    Partition: 2    Leader: 2    Replicas: 2    Isr: 2

 3. 使用kafka-console-producer.sh脚本生产消息 生产10条消息

bin/kafka-console-producer.sh --broker-list localhost:9092 --topic consumer_offsets_test

依次输入10条信息 ^C 退出

4.验证消息是否生产成功

bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list localhost:9092 --topic consumer_offsets_test --time -1
consumer_offsets_test:0:3
consumer_offsets_test:1:4
consumer_offsets_test:2:4 

 这里我生产了11条信息，当时kafka-console-producer.sh 多生产了一条

5.创建消费组 消费信息

bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic consumer_offsets_test --from-beginning --group consumer_offsets_group

消费完信息 ^C 退出

记住刚才创建consumer_offsets_group 消费组ID 稍后会用到

6. 查询consumer_offsets_group 在 __consumer_offsets topic 中存放的位移信息
__consumer_offsets 默认分区50 那么我们的consumer_offsets_group的位移信息是放到那个分区里面了呢？

通过如下公式即可获取：
Math.abs("consumer_offsets_group".hashCode()) % 50

结果放在了11分区上

7.获取对应分区的位移信息
 在获取__consumer_offsets 信息之前需要设置consumer.properties中exclude.internal.topics=false
 因为__consumer_offsets 属于内置 topic

bin/kafka-console-consumer.sh --topic __consumer_offsets --partition 11 --bootstrap-server localhost:9092 --formatter "kafka.coordinator.group.GroupMetadataManager\$OffsetsMessageFormatter" --consumer.config config/consumer.properties --from-beginning

[consumer_offsets_group/*消费组ID*/,consumer_offsets_test/*topic*/,1/*partition*/]::OffsetAndMetadata(offset=4/*提交的消费位移信息*/, leaderEpoch=Optional.empty, metadata=, commitTimestamp=1543567461031, expireTimestamp=None)
[consumer_offsets_group,consumer_offsets_test,0]::OffsetAndMetadata(offset=3, leaderEpoch=Optional.empty, metadata=, commitTimestamp=1543567461031, expireTimestamp=None)
[consumer_offsets_group,consumer_offsets_test,2]::OffsetAndMetadata(offset=4, leaderEpoch=Optional.empty, metadata=, commitTimestamp=1543567461031, expireTimestamp=None)
[consumer_offsets_group,consumer_offsets_test,1]::OffsetAndMetadata(offset=4, leaderEpoch=Optional.empty, metadata=, commitTimestamp=1543567466027, expireTimestamp=None)
[consumer_offsets_group,consumer_offsets_test,0]::OffsetAndMetadata(offset=3, leaderEpoch=Optional.empty, metadata=, commitTimestamp=1543567466027, expireTimestamp=None)
[consumer_offsets_group,consumer_offsets_test,2]::OffsetAndMetadata(offset=4, leaderEpoch=Optional.empty, metadata=, commitTimestamp=1543567466027, expireTimestamp=None)
[consumer_offsets_group,consumer_offsets_test,1]::OffsetAndMetadata(offset=4, leaderEpoch=Optional.empty, metadata=, commitTimestamp=1543567468638, expireTimestamp=None)
[consumer_offsets_group,consumer_offsets_test,0]::OffsetAndMetadata(offset=3, leaderEpoch=Optional.empty, metadata=, commitTimestamp=1543567468638, expireTimestamp=None)
[consumer_offsets_group,consumer_offsets_test,2]::OffsetAndMetadata(offset=4, leaderEpoch=Optional.empty, metadata=, commitTimestamp=1543567468638, expireTimestamp=None)