flume sink hdfs异常

1、异常消息如下:

016-08-26 14:19:17,704 (hdfs-sink1-call-runner-2) [ERROR - org.apache.flume.sink.hdfs.AbstractHDFSWriter.hflushOrSync(AbstractHDFSWriter.java:267)] Error while trying to hflushOrSync!
2016-08-26 14:19:17,704 (SinkRunner-PollingRunner-DefaultSinkProcessor) [WARN - org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:455)] HDFS IO error
java.io.IOException: Callable timed out after 10000 ms on file: hdfs://cluster/data01/kafka2hdfs/REMOTE_LOGIN_VERIFICATIOON_ONLINE/ds=2016-08-26/FlumeData.1472140808764.tmp
        at org.apache.flume.sink.hdfs.BucketWriter.callWithTimeout(BucketWriter.java:693)
        at org.apache.flume.sink.hdfs.BucketWriter.doFlush(BucketWriter.java:459)
        at org.apache.flume.sink.hdfs.BucketWriter.flush(BucketWriter.java:421)
        at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:442)
        at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
        at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.util.concurrent.TimeoutException
        at java.util.concurrent.FutureTask.get(FutureTask.java:201)
        at org.apache.flume.sink.hdfs.BucketWriter.callWithTimeout(BucketWriter.java:686)




2、发现flume配置实时三个参数被配置0了:

tier1.sinks.sink1.hdfs.rollInterval = 0
tier1.sinks.sink1.hdfs.rollSize = 0
tier1.sinks.sink1.hdfs.rollCount = 0


3、修改后的配置:

tier1.sinks.sink1.hdfs.path = hdfs://cluster/data01/kafka2hdfs/%{topic}/ds=%Y-%m-%d
tier1.sinks.sink1.hdfs.rollInterval = 60
tier1.sinks.sink1.hdfs.rollSize = 10240
tier1.sinks.hdfs-write.hdfs.batchSize = 100
tier1.sinks.sink1.hdfs.rollCount = 100
tier1.sinks.sink1.hdfs.fileType = DataStream
tier1.sinks.sink1.channel = channel1


评论 1
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值