ERROR flume.SinkRunner: Unable to deliver event. Exception follows. org.apache.flume.EventDeliveryEx

TTXS123456789ABC

于 2024-02-02 14:43:20 发布

阅读量630

点赞数 8

分类专栏： # 4-离线数仓-A级文章标签： flume apache python

本文链接：https://blog.csdn.net/qq_45972323/article/details/135993594

版权

4-离线数仓-A级专栏收录该内容

32 篇文章 0 订阅

订阅专栏

文章描述了在使用ApacheFlume进行数据增量同步时遇到的问题，错误提示未能正确解析事件中的timestamp头，导致无法进行时间基桶化。解决方法可能是检查TimestampInterceptor配置以确保正确填充该字段。

摘要由CSDN通过智能技术生成

Hadoop104

flume 增量同步报错



2024-02-02 14:31:36,123 (Log-BackgroundWorker-c1) [INFO - org.apache.flume.channel.file.EventQueueBackingStoreFile.beginCheckpoint(EventQueueBackingStoreFile.java:230)] Start checkpoint for /opt/module/flume/checkpoint/behavior2/checkpoint, elements to sync = 1
2024-02-02 14:31:36,125 (Log-BackgroundWorker-c1) [INFO - org.apache.flume.channel.file.EventQueueBackingStoreFile.checkpoint(EventQueueBackingStoreFile.java:255)] Updating checkpoint metadata: logWriteOrderID: 1706855257543, queueSize: 4833, queueHead: 999998
2024-02-02 14:31:36,132 (Log-BackgroundWorker-c1) [INFO - org.apache.flume.channel.file.Log.writeCheckpoint(Log.java:1065)] Updated checkpoint for file: /opt/module/flume/data/behavior2/log-6 position: 886110 logWriteOrderID: 1706855257543
2024-02-02 14:31:36,133 (Log-BackgroundWorker-c1) [INFO - org.apache.flume.channel.file.LogFile$RandomReader.close(LogFile.java:520)] Closing RandomReader /opt/module/flume/data/behavior2/log-1
2024-02-02 14:31:36,140 (Log-BackgroundWorker-c1) [INFO - org.apache.flume.channel.file.LogFile$RandomReader.close(LogFile.java:520)] Closing RandomReader /opt/module/flume/data/behavior2/log-2
2024-02-02 14:31:36,856 (SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR - org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:459)] process failed
java.lang.RuntimeException: Flume wasn't able to parse timestamp header in the event to resolve time based bucketing. Please check that you're correctly populating timestamp header (for example using TimestampInterceptor source interceptor).
	at org.apache.flume.formatter.output.BucketPath.replaceShorthand(BucketPath.java:258)
	at org.apache.flume.formatter.output.BucketPath.escapeString(BucketPath.java:460)
	at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:379)
	at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:67)
	at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:145)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NumberFormatException: For input string: "user_info"
	at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
	at java.lang.Long.parseLong(Long.java:589)
	at java.lang.Long.valueOf(Long.java:803)
	at org.apache.flume.formatter.output.BucketPath.replaceShorthand(BucketPath.java:253)
	... 5 more
2024-02-02 14:31:36,856 (SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR - org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:158)] Unable to deliver event. Exception follows.
org.apache.flume.EventDeliveryException: java.lang.RuntimeException: Flume wasn't able to parse timestamp header in the event to resolve time based bucketing. Please check that you're correctly populating timestamp header (for example using TimestampInterceptor source interceptor).
	at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:464)
	at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:67)
	at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:145)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.RuntimeException: Flume wasn't able to parse timestamp header in the event to resolve time based bucketing. Please check that you're correctly populating timestamp header (for example using TimestampInterceptor source interceptor).
	at org.apache.flume.formatter.output.BucketPath.replaceShorthand(BucketPath.java:258)
	at org.apache.flume.formatter.output.BucketPath.escapeString(BucketPath.java:460)
	at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:379)
	... 3 more
Caused by: java.lang.NumberFormatException: For input string: "user_info"
	at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
	at java.lang.Long.parseLong(Long.java:589)
	at java.lang.Long.valueOf(Long.java:803)
	at org.apache.flume.formatter.output.BucketPath.replaceShorthand(BucketPath.java:253)
	... 5 more