Exception in thread "StreamThread-1" org.apache.kafka.streams.errors.StreamsException: Input record ConsumerRecord(topic = log, partition = 0, offset = 10, CreateTime = -1, checksum = 2479511219, serialized key size = -1, serialized value size = 232, key = null, value = [B@66d1a849) has invalid (negative) timestamp. Possibly because a pre-0.10 producer client was used to write this record to Kafka without embedding a timestamp, or because the input topic was created before upgrading the Kafka cluster to 0.10+. Use a different TimestampExtractor to process this data.
at org.apache.kafka.streams.processor.FailOnInvalidTimestamp.onInvalidTimestamp(FailOnInvalidTimestamp.java:62)
at org.apache.kafka.streams.processor.ExtractRecordMetadataTimestamp.extract(ExtractRecordMetadataTimestamp.java:60)
at org.apache.kafka.streams.processor.FailOnInvalidTimestamp.extract(FailOnInvalidTimestamp.java:46)
at org.apache.kafka.streams.processor.internals.RecordQueue.addRawRecords(RecordQueue.java:86)
at org.apache.kafka.streams.processor.internals.PartitionGroup.addRawRecords(PartitionGroup.java:117)
at org.apache.kafka.streams.processor.internals.StreamTask.addRecords(StreamTask.java:158)
at org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:605)
at org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:361)
kafkastream作为消费者,flume作为生产者,,根据异常的意思是说消费者记录是有无效的时间戳,可能以为0.10前版本的生产者客户端写记录时没有内嵌时间戳(这里用的flume做生产者不存在这个问题,排除);或者因为输入主题是在将Kafka集群升级到0.10之前创建的+(我也不懂为啥,kafka_2.11_0.11版本有这问题,还望大佬指教),他也提供了解决方法使用一个 TimestampExtractor
解决
import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.apache.kafka.streams.processor.TimestampExtractor;
public class MyEventTimeExtractor implements TimestampExtractor{
@Override
public long extract(ConsumerRecord<Object, Object> record, long previousTimestamp) {
return System.currentTimeMillis();
}
}
并在
Properties properties = new Properties();
properties.put(StreamsConfig.TIMESTAMP_EXTRACTOR_CLASS_CONFIG,MyEventTimeExtractor.class.getName());