一:offset保存位置
kafka的offset管理,可以交给zookeeper实现,经过配置offsets.storage=zookeeper;也可以直接跟kafka自己管理,在你 的配置文件设置下面的配置项;
offsets.storage=kafka
dual.commit.enabled=true
kafka自己管理自己的offset,并经过命令可以查看
bin/kafka-consumer-groups.sh --bootstrap-server broker1:9092 --describe --group kafkatest
如上图可以查看到group中每个消费者对应topic的patition,当前消费的offset;每个partition中最大的offset等信息;经过测试,当没有消费者进行消费时候,若执行上面命令会出现下面的现象:消费者不见了
以上就是用kafka命令获取的结果;
下面是经过查看借鉴源码,用Java API实现的offset管理(kafka 版本为0.10)
<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka_2.10</artifactId>
<version>0.10.2.1</version>
</dependency>
<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka-clients</artifactId>
<version>0.10.2.1</version>
</dependency>
测试类代码如下:
public static void main(String[] args) throws InstantiationException, IllegalAccessException {
Properties props = new Properties();
props.put(CommonClientConfigs.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
AdminClient adminClient = AdminClient.create(props);
ConsumerGroupSummary consumerGroupSummary = adminClient.describeConsumerGroup("kafkatest");
if(consumerGroupSummary.state().equals("Empty")){
System.out.println("niaho");
}
Option<List<ConsumerSummary>> consumerSummaryOption = consumerGroupSummary.consumers();
List<ConsumerSummary> ConsumerSummarys = consumerSummaryOption.get();//获取组中的消费者
KafkaConsumer consumer = getNewConsumer();
for(int i=0;i<ConsumerSummarys.size();i++){ //循环组中的每一个消费者
ConsumerSummary consumerSummary = ConsumerSummarys.apply(i);
String consumerId = consumerSummary.consumerId();//获取消费者的id
scala.collection.immutable.Map<TopicPartition, Object> maps =
adminClient.listGroupOffsets("kafkatest");//或者这个组消费的所有topic,partition和当前消费到的offset
List<TopicPartition> topicPartitions= consumerSummary.assignment();//获取这个消费者下面的所有topic和partion
for(int j =0;j< topicPartitions.size();j++){ //循环获取每一个topic和partion
TopicPartition topicPartition = topicPartitions.apply(j);
String CURRENToFFSET = maps.get(topicPartition).get().toString();
long endOffset =getLogEndOffset(topicPartition);
System.out.println("topic的名字为:"+topicPartition.topic()+"====分区为:"+topicPartition.partition()+"===目前消费offset为:"+CURRENToFFSET+"===,此分区最后offset为:"+endOffset);
}
}
}
引用的两个方法代码如下:
获取消费者
public static KafkaConsumer getNewConsumer(){
Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("group.id", "kafkatest");
props.put("enable.auto.commit", "true");
props.put("auto.offset.reset", "earliest");
props.put("auto.commit.interval.ms", "1000");
props.put("auto.commit.interval.ms", "1000");
props.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
props.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props);
return consumer;
}
获取最后的offset:
public static long getLogEndOffset(TopicPartition topicPartition){
KafkaConsumer<String, String> consumer= getNewConsumer();
consumer.assign(Arrays.asList(topicPartition));
consumer.seekToEnd(Arrays.asList(topicPartition));
long endOffset = consumer.position(topicPartition);
return endOffset;
}
运行后,可以获取结果如下: