一.解决Flume执行命令时遇到的ChannelException和InterruptedException异常
1.问题描述:
在Flume的日志中,我们发现了如下异常信息:
2023-10-12 23:49:45,863 EFRCR scurce.ExecSource: Failed while running command: tail -F /opt/server/projectl.py
org.apache.flume.ChannelException: java.lang.InterruptedException at.org.apache.flume.channel.BasicTransactionSemantics.commit(BasicTransactionSemantics.java:154)
at.org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:19)
at.org.apache.flume.source.ExecSource$ExecRunnable.flushEventBatch(ExecSource.java:385)
at org.apache.flume.source.ExecSource$ExecRunnable.run(ExecSource.java:352)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
2.问题原因:
这个异常的出现通常是由于以下原因引起的:
(1).资源不足:系统资源(例如内存或CPU)不足,导致Flume执行过程中被迫中断。
3.解决方法:
针对上述问题,我们可以采取以下措施来解决Flume遇到的ChannelException和InterruptedException异常:
(1).增加系统资源:如果系统资源不足,可以考虑增加系统的内存或CPU。确保Flume在运行时有 足够的资源支持。把上述图片1000改为2000
二.解决Flume执行命令时遇到的时间戳的问题
1.问题描述:
在Flume的日志中,我们发现了如下异常信息:
java.lang.NullPointerException: Expected timestamp in the Flume event headers, but it was null
at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:204)
at org.apache.flume.formatter.output.BucketPath.replaceShorthand(BucketPath.java:200)
at org.apache.flume.formatter.output.BucketPath.escapeString(BucketPath.java:396)
at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:388)
at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
at java.lang.Thread.run(Thread.java:744)
2023-10-12 15:31:41,836 (SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR - org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:160)] Unable to deliver event. Exception follows.
org.apache.flume.EventDeliveryException: java.lang.NullPointerException: Expected timestamp in the Flume event headers, but it was null
at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:471)
at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
at java.lang.Thread.run(Thread.java:744)
Caused by: java.lang.NullPointerException: Expected timestamp in the Flume event headers, but it was null
at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:204)
at org.apache.flume.formatter.output.BucketPath.replaceShorthand(BucketPath.java:200)
at org.apache.flume.formatter.output.BucketPath.escapeString(BucketPath.java:396)
at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:388)
... 3 more
2.问题原因:
这个异常的出现通常是由于以下原因引起的:
(1).使用Flume的HDFSEventSink时,Flume事件中缺少预期的时间戳字段。
3.解决方法:
(1).加入下面代码:
a1.sinks.k1.hdfs.useLocalTimeStamp = true
a1.sinks.k1.hdfs.fileType = DataStream
三.解决Flume执行命令时遇到的guava-11.0.2.jar的版本不兼容
1.问题描述:
在Flume的日志中,我们发现了如下异常信息:
2.问题原因:
这个异常的出现通常是由于以下原因引起的:
(1).flume安装目录中lib文件夹下的guava-11.0.2.jar的版本不兼容。
3.解决方法:
(1).删除guava包
[user@hadoop102 lib]$ rm /opt/module/flume/lib/guava-11.0.2.jar