不同版本的kafka手动设置consumer的offset

版本 < 0.9.0.0

• 旧版本中消费者的offset是托管在zookeeper中,目录/consumers/[group_id]/offsets/[topic]/[broker_id-part_id]下, 修改offset的话,需要手动修改子目录中指定group_id的offset值就可以了
• 前提条件: 关闭所有的消费者
• List all topics
  kafka-topics --list --zookeeper localhost:2181
• Get Offsets for the topic
  kafka-run-class kafka.tools.GetOffsetShell --broker-list localhost:9092 -topic vital_signs --time -1
• Set the offset manually

$ ZOOKEEPER_HOME/zkCli
$ ls /consumers/flasfka/offsets/vital_signs

$ set /consumers/flasfka/offsets/vital_signs/0 11492949

• Verify the new offset
  kafka-run-class kafka.tools.GetOffsetShell --broker-list localhost:9092 -topic vital_signs --time -1

以上命令来自: 连接

版本 > 0.9.0.0

• 前提条件: 关闭所有的消费者
• Get Offsets for the topic
  bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list BORKER_HOST1:PORT1,BORKER_HSOT2:PORT2 --topic TOPIC_NAME --time -2
• Set the offset manually
  bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --group you_consumer_group_name --topic you_topic_name --execute --reset-offsets --to-offset 80
• Verify the new offset
  bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list BORKER_HOST1:PORT1,BORKER_HSOT2:PORT2 --topic TOPIC_NAME --time -2
• 用java客户端修改

Properties _prop = PropUtils.getProp();
Properties props = new Properties();
props.put("bootstrap.servers", _prop.getProperty("bootstrap.servers"));
props.put("group.id", group_id);
props.put("enable.auto.commit", "false");
props.put("fetch.message.max.bytes", _prop.getProperty("fetch.message.max.bytes"));
props.put("auto.commit.interval.ms", "1000");
props.put("session.timeout.ms", "30000");
props.put("key.deserializer", StringDeserializer.class.getName());
props.put("value.deserializer", StringDeserializer.class.getName());
KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props)
//        consumer.subscribe(Arrays.asList(topic));

String topic = "uoamp_topic";
List<PartitionInfo> partitionInfos = consumer.partitionsFor("uoamp_topic");
List<TopicPartition> partitions = partitionInfos.stream()
        .map(info -> new TopicPartition(topic, info.partition()))
        .collect(Collectors.toList());
consumer.assign(partitions);
partitions.forEach(p -> System.out.println(p.toString() + ": " + consumer.position(p)));

Map<TopicPartition, OffsetAndMetadata> offsetMap = new HashMap<>();
for (TopicPartition p : partitions) {
    offsetMap.put(p, new OffsetAndMetadata(1));
}
consumer.commitSync(offsetMap);

Thread.sleep(3000);

partitions.forEach(p -> System.out.println(p.toString() + ": " + consumer.position(p)));

consumer.subscribe(Arrays.asList(topic))和consumer.assign(partitions)只用一个,只用做消费者用前者即可.