kafka 的 auto.offset.reset 含义详解

最近也是有人问我kafka的auto.offset.reset设置为earliest后怎么结果和自己想象的不一样呢,相信很多人都对这个参数心存疑惑,今天来详细讲解一下:

kafka-0.10.1.X版本之前: auto.offset.reset 的值为smallest,和,largest.(offest保存在zk中)

kafka-0.10.1.X版本之后: auto.offset.reset 的值更改为:earliest,latest,和none (offest保存在kafka的一个特殊的topic名为:__consumer_offsets里面)

顾名思义,earliest就是从最开始消费数据,latest即为从最新的数据开始消费,但我们在使用的时候发现并不是这样的.下面就来详细测试一下.

先看一下官网对auto.offset.reset的解释吧(测试版本为0.10.2.1)

这个解释也比较的抽象,不太好理解,所以我们还是自己动手去验证一下吧.

新建一个topic,名为jason_1122,先查看一下这个topic的信息:

Topic:jason_1122	PartitionCount:10	ReplicationFactor:3	Configs:
	Topic: jason_1122	Partition: 0	Leader: 3	Replicas: 3,2,1	Isr: 3,2,1
	Topic: jason_1122	Partition: 1	Leader: 1	Replicas: 1,3,2	Isr: 1,3,2
	Topic: jason_1122	Partition: 2	Leader: 2	Replicas: 2,1,3	Isr: 2,1,3
	Topic: jason_1122	Partition: 3	Leader: 3	Replicas: 3,1,2	Isr: 3,1,2
	Topic: jason_1122	Partition: 4	Leader: 1	Replicas: 1,2,3	Isr: 1,2,3
	Topic: jason_1122	Partition: 5	Leader: 2	Replicas: 2,3,1	Isr: 2,3,1
	Topic: jason_1122	Partition: 6	Leader: 3	Replicas: 3,2,1	Isr: 3,2,1
	Topic: jason_1122	Partition: 7	Leader: 1	Replicas: 1,3,2	Isr: 1,3,2
	Topic: jason_1122	Partition: 8	Leader: 2	Replicas: 2,1,3	Isr: 2,1,3
	Topic: jason_1122	Partition: 9	Leader: 3	Replicas: 3,1,2	Isr: 3,1,2

可以看到jason_1122这个topic有10分分区,3分副本.

测试的过程:

1.先写入10条数据

2.设置offest为false,分别测试earliest,latest,和none,这三种情况

3,设置offest为true,分别测试earliest,latest,和none,这三种情况

3.在写入10条数据

4.设置offest为false,分别测试earliest,latest,和none,这三种情况

5.设置offest为true,分别测试earliest,latest,和none,这三种情况

测试一:

先写入10条数据, 设置自动提交offest为false,然后分别测试auto.offset.reset设置为earliest,latest,和none的情况.

测试结果:

(1).当auto.offset.reset为earliest时,消费到了10条数据如下:

0--null--hello jason what are you doing--1542886948685--6
0--null--hello jason what are you doing--1542886948685--3
0--null--hello jason what are you doing--1542886948686--0
0--null--hello jason what are you doing--1542886948658--8
0--null--hello jason what are you doing--1542886948684--7
0--null--hello jason what are you doing--1542886948683--5
0--null--hello jason what are you doing--1542886948684--4
0--null--hello jason what are you doing--1542886948683--2
0--null--hello jason what are you doing--1542886948685--1

(2).当auto.offset.reset为latest时,没有消费到数据.

(3).当auto.offset.reset为none时,抛出异常如下:

Exception in thread "main" org.apache.kafka.clients.consumer.NoOffsetForPartitionException: Undefined offset with no reset policy for partition: jason_1122-4
	at org.apache.kafka.clients.consumer.internals.Fetcher.resetOffset(Fetcher.java:369)
	at org.apache.kafka.clients.consumer.internals.Fetcher.updateFetchPositions(Fetcher.java:247)
	at org.apache.kafka.clients.consumer.KafkaConsumer.updateFetchPositions(KafkaConsumer.java:1602)
	at org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:1035)
	at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:995)
	at kafka.KafkaConsumer$.main(KafkaConsumer.scala:19)
	at kafka.KafkaConsumer.main(KafkaConsumer.scala)

测试二:

设置自动提交offest为true,然后分别测试auto.offset.reset设置为earliest,latest,和none的情况.

测试结果:

(1)当auto.offset.reset为earliest时,消费到了10条数据如下:

0--null--hello jason what are you doing--1542886948658--8
0--null--hello jason what are you doing--1542886948684--7
0--null--hello jason what are you doing--1542886948685--6
0--null--hello jason what are you doing--1542886948683--5
0--null--hello jason what are you doing--1542886948684--4
0--null--hello jason what are you doing--1542886948685--3
0--null--hello jason what are you doing--1542886948683--2
0--null--hello jason what are you doing--1542886948685--1
0--null--hello jason what are you doing--1542886948686--0

(2)当auto.offset.reset为latest时,没有消费到数据.

(3)当auto.offset.reset为none时,没有消费到数据,也没有抛出异常.

测试三:

在写入10条数据,一共为20条数据,设置自动提交offest为false,然后分别测试auto.offset.reset设置为earliest,latest,和none的情况.

测试结果:

(1)当auto.offset.reset为earliest时,消费到了10条数据如下:

1--null--hello jason what are you doing--1542888276632--8
1--null--hello jason what are you doing--1542888276631--7
1--null--hello jason what are you doing--1542888276631--5
1--null--hello jason what are you doing--1542888276631--4
1--null--hello jason what are you doing--1542888276607--2
1--null--hello jason what are you doing--1542888276631--1
0--null--hello jason what are you doing--1542888276631--9
1--null--hello jason what are you doing--1542888276631--6
1--null--hello jason what are you doing--1542888276631--3
1--null--hello jason what are you doing--1542888276632--0

(2)当auto.offset.reset为latest时,同样消费到了10条数据如下:

0--null--hello jason what are you doing--1542888276631--9
1--null--hello jason what are you doing--1542888276631--6
1--null--hello jason what are you doing--1542888276631--3
1--null--hello jason what are you doing--1542888276632--0
1--null--hello jason what are you doing--1542888276632--8
1--null--hello jason what are you doing--1542888276631--7
1--null--hello jason what are you doing--1542888276631--5
1--null--hello jason what are you doing--1542888276631--4
1--null--hello jason what are you doing--1542888276607--2
1--null--hello jason what are you doing--1542888276631--1

(3)当auto.offset.reset为none时,同样消费到了10条数据如下:

0--null--hello jason what are you doing--1542888276631--9
1--null--hello jason what are you doing--1542888276631--6
1--null--hello jason what are you doing--1542888276631--3
1--null--hello jason what are you doing--1542888276632--0
1--null--hello jason what are you doing--1542888276632--8
1--null--hello jason what are you doing--1542888276631--7
1--null--hello jason what are you doing--1542888276631--5
1--null--hello jason what are you doing--1542888276631--4
1--null--hello jason what are you doing--1542888276607--2
1--null--hello jason what are you doing--1542888276631--1

测试四:

设置自动提交offest为true,然后分别测试auto.offset.reset设置为earliest,latest,和none的情况.

测试结果:

(1)当auto.offset.reset为earliest时,消费到了10条数据如下:

0--null--hello jason what are you doing--1542888276631--9
1--null--hello jason what are you doing--1542888276631--6
1--null--hello jason what are you doing--1542888276631--3
1--null--hello jason what are you doing--1542888276632--0
1--null--hello jason what are you doing--1542888276632--8
1--null--hello jason what are you doing--1542888276631--7
1--null--hello jason what are you doing--1542888276631--5
1--null--hello jason what are you doing--1542888276631--4
1--null--hello jason what are you doing--1542888276607--2
1--null--hello jason what are you doing--1542888276631--1

(2)当auto.offset.reset为latest时,没有消费到数据

(3)当auto.offset.reset为none时,也没有消费到数据

根据上面的测试,得到最终的结论:

如果存在已经提交的offest时,不管设置为earliest 或者latest 都会从已经提交的offest处开始消费

如果不存在已经提交的offest时,earliest 表示从头开始消费,latest 表示从最新的数据消费,也就是新产生的数据.

none topic各分区都存在已提交的offset时，从提交的offest处开始消费；只要有一个分区不存在已提交的offset，则抛出异常

如果有写的不对的地方 欢迎大家指正 如果有什么疑问 可以加下面的技术交流群