Kafka AdminClient 管理Kafka Offset代码实现
一:offset保存位置
kafka的offset管理,可以交给zookeeper实现,经过配置offsets.storage=zookeeper;也可以直接跟kafka自己管理,在你 的配置文件设置下面的配置项;
offsets.storage=kafka
dual.commit.enabled=true
kafka自己管理自己的offset,并经过命令可以查看
bin/kafka-consumer-groups.sh --bootstrap-server broker1:9092 --describe --group kafkatest
如上图可以查看到group中每个消费者对应topic的patition,当前消费的offset;每个partition中最大的offset等信息;经过测试,当没有消费者进行消费时候,若执行上面命令会出现下面的现象:消费者不见了
以上就是用kafka命令获取的结果;
下面是经过查看借鉴源码,用Java API实现的offset管理(kafka 版本为0.10)
<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka_2.10</artifactId>
<version>0.10.2.1</version>
</dependency>
<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka-clients</artifactId>
<version>0.10.2.1</version>
</dependency>
测试类代码如下:
- public static void main(String[] args) throws InstantiationException, IllegalAccessException {
- Properties props = new Properties();
- props.put(CommonClientConfigs.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
- AdminClient adminClient = AdminClient.create(props);
- ConsumerGroupSummary consumerGroupSummary = adminClient.describeConsumerGroup("kafkatest");
- if(consumerGroupSummary.state().equals("Empty")){
- System.out.println("niaho");
- }
- Option<List<ConsumerSummary>> consumerSummaryOption = consumerGroupSummary.consumers();
- List<ConsumerSummary> ConsumerSummarys = consumerSummaryOption.get();//获取组中的消费者
- KafkaConsumer consumer = getNewConsumer();
- for(int i=0;i<ConsumerSummarys.size();i++){ //循环组中的每一个消费者
- ConsumerSummary consumerSummary = ConsumerSummarys.apply(i);
- String consumerId = consumerSummary.consumerId();//获取消费者的id
- scala.collection.immutable.Map<TopicPartition, Object> maps =
- adminClient.listGroupOffsets("kafkatest");//或者这个组消费的所有topic,partition和当前消费到的offset
- List<TopicPartition> topicPartitions= consumerSummary.assignment();//获取这个消费者下面的所有topic和partion
- for(int j =0;j< topicPartitions.size();j++){ //循环获取每一个topic和partion
- TopicPartition topicPartition = topicPartitions.apply(j);
- String CURRENToFFSET = maps.get(topicPartition).get().toString();
- long endOffset =getLogEndOffset(topicPartition);
- System.out.println("topic的名字为:"+topicPartition.topic()+"====分区为:"+topicPartition.partition()+"===目前消费offset为:"+CURRENToFFSET+"===,此分区最后offset为:"+endOffset);
- }
- }
- }
获取消费者
- public static KafkaConsumer getNewConsumer(){
- Properties props = new Properties();
- props.put("bootstrap.servers", "localhost:9092");
- props.put("group.id", "kafkatest");
- props.put("enable.auto.commit", "true");
- props.put("auto.offset.reset", "earliest");
- props.put("auto.commit.interval.ms", "1000");
- props.put("auto.commit.interval.ms", "1000");
- props.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
- props.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
- KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props);
- return consumer;
- }
- public static long getLogEndOffset(TopicPartition topicPartition){
- KafkaConsumer<String, String> consumer= getNewConsumer();
- consumer.assign(Arrays.asList(topicPartition));
- consumer.seekToEnd(Arrays.asList(topicPartition));
- long endOffset = consumer.position(topicPartition);
- return endOffset;
- }