KafkaStream消费Flume出现 has invalid (negative) timestamp 异常

Exception in thread "StreamThread-1" org.apache.kafka.streams.errors.StreamsException: Input record ConsumerRecord(topic = log, partition = 0, offset = 10, CreateTime = -1, checksum = 2479511219, serialized key size = -1, serialized value size = 232, key = null, value = [B@66d1a849) has invalid (negative) timestamp. Possibly because a pre-0.10 producer client was used to write this record to Kafka without embedding a timestamp, or because the input topic was created before upgrading the Kafka cluster to 0.10+. Use a different TimestampExtractor to process this data.
	at org.apache.kafka.streams.processor.FailOnInvalidTimestamp.onInvalidTimestamp(FailOnInvalidTimestamp.java:62)
	at org.apache.kafka.streams.processor.ExtractRecordMetadataTimestamp.extract(ExtractRecordMetadataTimestamp.java:60)
	at org.apache.kafka.streams.processor.FailOnInvalidTimestamp.extract(FailOnInvalidTimestamp.java:46)
	at org.apache.kafka.streams.processor.internals.RecordQueue.addRawRecords(RecordQueue.java:86)
	at org.apache.kafka.streams.processor.internals.PartitionGroup.addRawRecords(PartitionGroup.java:117)
	at org.apache.kafka.streams.processor.internals.StreamTask.addRecords(StreamTask.java:158)
	at org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:605)
	at org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:361)

kafkastream作为消费者,flume作为生产者,,根据异常的意思是说消费者记录是有无效的时间戳,可能以为0.10前版本的生产者客户端写记录时没有内嵌时间戳(这里用的flume做生产者不存在这个问题,排除);或者因为输入主题是在将Kafka集群升级到0.10之前创建的+(我也不懂为啥,kafka_2.11_0.11版本有这问题,还望大佬指教),他也提供了解决方法使用一个 TimestampExtractor

解决

import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.apache.kafka.streams.processor.TimestampExtractor;
 
 
 public class MyEventTimeExtractor implements TimestampExtractor{
 
 
 @Override
 public long extract(ConsumerRecord<Object, Object> record, long previousTimestamp) {
	 return System.currentTimeMillis();
 
 	}
 }

并在

Properties properties = new Properties(); 
properties.put(StreamsConfig.TIMESTAMP_EXTRACTOR_CLASS_CONFIG,MyEventTimeExtractor.class.getName());
  • 0
    点赞
  • 1
    收藏
    觉得还不错? 一键收藏
  • 0
    评论

“相关推荐”对你有帮助么?

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值