Flume安装和配置

最近，我在写一个开发文档，觉得有必要也上传到网上分享一下。仅供参考，如有差错，愿听修正。下面介绍Flume的安装和配置。

1 准备Flume-ng

到flume官网下载最新稳定版本的flume-ng，目前的版本是1.4.0。

下载地址：

http://mirrors.hust.edu.cn/apache/flume/1.4.0/apache-flume-1.4.0-bin.tar.gz

下载后，解压说文件。

tar –xvf apache-flume-1.4.0-bin.tar.gz

//重命名目录

mv apache-flume-1.4.0-bin flume

//移动到指定目录下,比如/opt/

mv flume /opt/

2 修改配置文件

2.1 配置log4.properties

这里，我默认采用flume提供的模板。

flume.root.logger=INFO,LOGFILE
flume.log.dir=./logs
flume.log.file=flume.log
 
log4j.logger.org.apache.flume.lifecycle= INFO
log4j.logger.org.jboss= WARN
log4j.logger.org.mortbay= INFO
log4j.logger.org.apache.avro.ipc.NettyTransceiver= WARN
log4j.logger.org.apache.hadoop= INFO
 
#Define the root logger to the system property "flume.root.logger".
log4j.rootLogger=${flume.root.logger}
 
 
#Stock log4j rolling file appender
#Default log rotation configuration
log4j.appender.LOGFILE=org.apache.log4j.RollingFileAppender
log4j.appender.LOGFILE.MaxFileSize=100MB
log4j.appender.LOGFILE.MaxBackupIndex=10
log4j.appender.LOGFILE.File=${flume.log.dir}/${flume.log.file}
log4j.appender.LOGFILE.layout=org.apache.log4j.PatternLayout
log4j.appender.LOGFILE.layout.ConversionPattern=%d{ddMMM yyyy HH:mm:ss,SSS} %-5p [%t] (%C.%M:%L) %x - %m%n
 
 
#Warning: If you enable the following appender it will fill up your disk if youdon't have a cleanup job!
#This uses the updated rolling file appender from log4j-extras that supports areliable time-based rolling policy.
#See http://logging.apache.org/log4j/companions/extras/apidocs/org/apache/log4j/rolling/TimeBasedRollingPolicy.html
#Add "DAILY" to flume.root.logger above if you want to use this
log4j.appender.DAILY=org.apache.log4j.rolling.RollingFileAppender
log4j.appender.DAILY.rollingPolicy=org.apache.log4j.rolling.TimeBasedRollingPolicy
log4j.appender.DAILY.rollingPolicy.ActiveFileName=${flume.log.dir}/${flume.log.file}
log4j.appender.DAILY.rollingPolicy.FileNamePattern=${flume.log.dir}/${flume.log.file}.%d{yyyy-MM-dd}
log4j.appender.DAILY.layout=org.apache.log4j.PatternLayout
log4j.appender.DAILY.layout.ConversionPattern=%d{ddMMM yyyy HH:mm:ss,SSS} %-5p [%t] (%C.%M:%L) %x - %m%n
 
 
#console
#Add "console" to flume.root.logger above if you want to use this
log4j.appender.console=org.apache.log4j.ConsoleAppender
log4j.appender.console.target=System.err
log4j.appender.console.layout=org.apache.log4j.PatternLayout
log4j.appender.console.layout.ConversionPattern=%d(%t) [%p - %l] %m%n

2.2 配置flume-env.sh文件

根据需用，你可以配置flume的环境变量，flume-ng启动会加载这些环境变量。有需要就配置，没有实用其它jar包就没有必要。

2.3 配置flume-ng配置加载文件

根据业务逻辑的需要配置flume的配置文件，只需要以properties结尾的文件就可以了。下面给出了一个文件flume-conf.properties的配置，source采用的是avro格式文件，channel采用memory，sink采用HDFS。

agent.sources=s1
agent.channels=c1
agent.sinks=k1
 
agent.sources.s1.type=avro
agent.sources.s1.channels=c1
agent.sources.s1.bind=0.0.0.0
agent.sources.s1.port=51515
agent.sources.s1.interceptors=i1
agent.sources.s1.interceptors.i1.type=timestamp
 
agent.channels.c1.type=memory
 
agent.sinks.k1.type=hdfs
agent.sinks.k1.channel=c1
agent.sinks.k1.hdfs.path=/flume/%Y/%m
agent.sinks.k1.hdfs.filePrefix=flume
agent.sinks.k1.hdfs.fileSuffix=.log
agent.sinks.k1.hdfs.rollInterval=3600
agent.sinks.k1.hdfs.rollCount=0
agent.sinks.k1.hdfs.rollSize=0
agent.sinks.k1.hdfs.fileType=DataStream
agent.sinks.k1.hdfs.writeFormat=Text
agent.sinks.k1.hdfs.useLocalTimeStamp=true

进入flume的主目录，运行下面脚本就可以启动一个flume-ng agnet。

bin/flume-ngagent -n agent -c conf -f conf/flume-conf.properties-Dflume.root.logger=DEBUG,console