1、异常消息如下:
016-08-26 14:19:17,704 (hdfs-sink1-call-runner-2) [ERROR - org.apache.flume.sink.hdfs.AbstractHDFSWriter.hflushOrSync(AbstractHDFSWriter.java:267)] Error while trying to hflushOrSync!
2016-08-26 14:19:17,704 (SinkRunner-PollingRunner-DefaultSinkProcessor) [WARN - org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:455)] HDFS IO error
java.io.IOException: Callable timed out after 10000 ms on file: hdfs://cluster/data01/kafka2hdfs/REMOTE_LOGIN_VERIFICATIOON_ONLINE/ds=2016-08-26/FlumeData.1472140808764.tmp
at org.apache.flume.sink.hdfs.BucketWriter.callWithTimeout(BucketWriter.java:693)
at org.apache.flume.sink.hdfs.BucketWriter.doFlush(BucketWriter.java:459)
at org.apache.flume.sink.hdfs.BucketWriter.flush(BucketWriter.java:421)
at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:442)
at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.util.concurrent.TimeoutException
at java.util.concurrent.FutureTask.get(FutureTask.java:201)
at org.apache.flume.sink.hdfs.BucketWriter.callWithTimeout(BucketWriter.java:686)
2、发现flume配置实时三个参数被配置0了:
tier1.sinks.sink1.hdfs.rollInterval = 0
tier1.sinks.sink1.hdfs.rollSize = 0
tier1.sinks.sink1.hdfs.rollCount = 0
3、修改后的配置:
tier1.sinks.sink1.hdfs.path = hdfs://cluster/data01/kafka2hdfs/%{topic}/ds=%Y-%m-%d
tier1.sinks.sink1.hdfs.rollInterval = 60
tier1.sinks.sink1.hdfs.rollSize = 10240
tier1.sinks.hdfs-write.hdfs.batchSize = 100
tier1.sinks.sink1.hdfs.rollCount = 100
tier1.sinks.sink1.hdfs.fileType = DataStream
tier1.sinks.sink1.channel = channel1