flume 中的 hdfs sink round 和roll

最新推荐文章于 2022-11-01 22:26:23 发布

weixin_34274029

最新推荐文章于 2022-11-01 22:26:23 发布

阅读量268

点赞数

文章标签：大数据

http://blog.csdn.net/kntao/article/details/49278239

http://flume.apache.org/FlumeUserGuide.html#exec-source

默认的是是SequenceFile所以数据存在hdfs上通过命令查看的时候会是乱码,如果此时需要修改filetype和writeFormat来修改

hdfs.fileType SequenceFile File format: currently SequenceFile, DataStream or CompressedStream (1)DataStream will not compress output file and please don’t set codeC (2)CompressedStream requires set hdfs.codeC with an available codeC

其中DataStream是文本文件

a1.sinks.k1.type = hdfs
a1.sinks.k1.channel = c1
a1.sinks.k1.hdfs.path = /maats/%{APPSTORE}/%Y%m%d/%H
a1.sinks.k1.hdfs.filePrefix = events-
a1.sinks.k1.hdfs.round = true
a1.sinks.k1.hdfs.roundValue = 10
a1.sinks.k1.hdfs.roundUnit = minute
a1.sinks.k1.hdfs.useLocalTimeStamp = true
a1.sinks.k1.hdfs.fileType=DataStream
a1.sinks.k1.hdfs.writeFormat=Text