flume监听文件目录sink到hdfs配置

一：flume介绍

Flume是一个分布式、可靠、和高可用的海量日志聚合的系统，支持在系统中定制各类数据发送方，用于收集数据；同时Flume提供对数据进行简单处理，并写到各种数据接受方（可定制）的能力。

Flume架构分为三个部分 源-Source，接收器-Sink，通道-Channel。

二：配置文件

此配置文件source为一个目录，注意，该目录下的文件应为只读，不可写，且文件名不能相同，采用的channels为file，sink为hdfs，此处往hdfs写的策略是当时间达到3600s或者文件大小达到128M。

agent1.sources = spooldirSource

agent1.channels = fileChannel

agent1.sinks = hdfsSink

agent1.sources.spooldirSource.type=spooldir

agent1.sources.spooldirSource.spoolDir=/data/lwq/new_log

agent1.sources.spooldirSource.channels=fileChannel

agent1.sinks.hdfsSink.type=hdfs

agent1.sinks.hdfsSink.hdfs.path=hdfs://dev228:8020/raw/lwq/%y-%m-%d

agent1.sinks.hdfsSink.hdfs.filePrefix=lwq

agent1.sinks.sink1.hdfs.round = true

# Number of seconds to wait before rolling current file (0 = never

roll based on time interval)

agent1.sinks.hdfsSink.hdfs.rollInterval = 3600

# File size to trigger roll, in bytes (0: never roll based on file

size)

agent1.sinks.hdfsSink.hdfs.rollSize = 128000000

agent1.sinks.hdfsSink.hdfs.rollCount = 0

agent1.sinks.hdfsSink.hdfs.batchSize = 1000

#Rounded down to the highest multiple of this (in the unit configured

using hdfs.roundUnit), less than current time.

agent1.sinks.hdfsSink.hdfs.roundValue = 1

agent1.sinks.hdfsSink.hdfs.roundUnit = minute

agent1.sinks.hdfsSink.hdfs.useLocalTimeStamp = true

agent1.sinks.hdfsSink.channel=fileChannel

agent1.sinks.hdfsSink.hdfs.fileType = DataStream

agent1.channels.fileChannel.type = file

agent1.channels.fileChannel.checkpointDir=/usr/share/apache-flume-

1.5.0-bin/checkpoint

agent1.channels.fileChannel.dataDirs=/usr/share/apache-flume-1.5.0-

bin/dataDir

三：启动命令

1    cd /opt/flume-1.6.0

2   ./bin/flume-ng agent --conf ./conf/ -f conf/flume.conf

     -Dflume.root.logger=DEBUG,console -n agent1 > log.log 2>&1 &