flume--收集日志到hdfs

最新推荐文章于 2022-09-21 00:18:30 发布

wangfutai91

最新推荐文章于 2022-09-21 00:18:30 发布

阅读量322

点赞数

分类专栏： flume

本文链接：https://blog.csdn.net/suojie123/article/details/86577779

版权

flume 专栏收录该内容

11 篇文章 0 订阅

订阅专栏

例子：
a1.sources = s1
a1.channels=c1
a1.sinks=k1
#tcp协议
a1.sources.s1.type =  syslogtcp
a1.sources.s1.port= 5140
a1.sources.s1.host= wangfutai
a1.sources.s1.channels = c1

a1.channels = c1
a1.channels.c1.type = SPILLABLEMEMORY 
a1.channels.c1.memoryCapacity = 10000 
a1.channels.c1.overflowCapacity = 1000000 
a1.channels.c1.byteCapacity = 800000 
a1.channels.c1.checkpointDir =/home/wangfutai/a/flume/checkPoint
a1.channels.c1.dataDirs = /home/wangfutai/a/flume/data
#必须写
a1.sinks = k1
a1.sinks.k1.type = hdfs 
a1.sinks.k1.channel = c1
a1.sinks.k1.hdfs.path =/user/wangfutai/flume/%y_%m_%d/%H_%M_%S 
#文件的名称
a1.sinks.k1.hdfs.filePrefix = wangfutai- 
a1.sinks.k1.hdfs.fileSuffix=.flu
a1.sinks.k1.hdfs.inUsePrefix =inUse-
a1.sinks.k1.hdfs.inUseSuffix =.t
#目录回滚开启后每10分钟回滚一次
a1.sinks.k1.hdfs.round = true 
a1.sinks.k1.hdfs.roundValue = 10 
a1.sinks.k1.hdfs.roundUnit = minute
#日志回滚
#定时回滚 收集的文件 每隔60s生成一个文本
#配置滚动生成
a1.sinks.k1.hdfs.rollInterval = 60
a1.sinks.k1.hdfs.rollSize = 0
a1.sinks.k1.hdfs.rollCount = 0
a1.sinks.k1.hdfs.idleTimeout = 0
#输出格式
a1.sinks.k1.hdfs.fileType =  DataStream
a1.sinks.k1.hdfs.writeFormat = Text
#指定timeZone
a1.sinks.k1.hdfs.useLocalTimeStamp = true
a1.sinks.k1.hdfs.timeZone  = Asia/Shanghai