kafka指定时间戳timestamp消费

kafka指定时间戳消费


代码如下

代码如下(示例):

/**
 * @author xixingya
 * @version 1.0
 * @date 2021/9/16 2:08 PM
 */
@Slf4j
@Data
public class KafkaTimeStampConsumer {

    private KafkaConsumer<String, String> kafkaConsumer;

    public KafkaTimeStampConsumer(String topic,Long timestamp,String cluster){
        Map<String, Object> props = Maps.newHashMap();

        //集群地址
        props.put("bootstrap.servers", cluster);

        //设置我们独特的消费者的组id
        props.put("group.id", "mygroup" );
        //设置手动提交
        props.put("enable.auto.commit", "false");
        //这个可以设置大一点
        props.put("session.timeout.ms", "30000");

        //props.put("auto.offset.reset", "earliest");

        props.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
        props.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");

        //我一般测试单条报错数据,1
        props.put("max.poll.records", 1500);
        kafkaConsumer = new KafkaConsumer<>(props);
        Map<TopicPartition,Long> map =new HashMap<>();
        final List<PartitionInfo> partitionInfos = kafkaConsumer.partitionsFor(topic);
        for (PartitionInfo partitionInfo : partitionInfos) {
            map.put(new TopicPartition(topic,partitionInfo.partition()),timestamp);
        }
        final Map<TopicPartition, OffsetAndTimestamp> timestampMap = kafkaConsumer.offsetsForTimes(map);

        List<TopicPartition> topicPartitionList = new ArrayList<>(timestampMap.keySet());
        kafkaConsumer.assign(topicPartitionList);
        timestampMap.forEach((topicPartition, offsetAndTimestamp) -> {
            if(offsetAndTimestamp!=null){
                kafkaConsumer.seek(topicPartition,offsetAndTimestamp.offset());
                int partition = topicPartition.partition();
                long timestamps =offsetAndTimestamp.timestamp();
                long offset = offsetAndTimestamp.offset();
                log.info("[KafkaTimeStampConsumer] partition = {} timestamp={} offset = {}",partition,timestamps,offset);
            }
        });
    }

    public ConsumerRecords<String, String> consume(){
        return kafkaConsumer.poll(36000);
    }

    public void close(){
        log.info("[KafkaTimeStampConsumer] close topic = {}", JSON.toJSONString(kafkaConsumer.listTopics()));
        kafkaConsumer.close();
    }


    public static void main(String[] args) {
        KafkaTimeStampConsumer kafkaTimeStampConsumer=new KafkaTimeStampConsumer("aaa",1632705050000L,"test-kafka.aaa.com:9092");
        while (true){
            final ConsumerRecords<String, String> records = kafkaTimeStampConsumer.kafkaConsumer.poll(36000);
            for (ConsumerRecord<String, String> record : records) {
                System.out.println("partition = "+record.partition()+"record = "+record+"offset = "+ record.offset());
                kafkaTimeStampConsumer.kafkaConsumer.commitAsync();
            }
        }
    }
}
  • 4
    点赞
  • 3
    收藏
    觉得还不错? 一键收藏
  • 0
    评论

“相关推荐”对你有帮助么?

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值