Flume安装配置

flume安装前需要搭建好hadoop平台

1.解压Flume安装包到/opt/module中

tar -zxvf apache-flume-1.11.0-bin.tar.gz -C /opt/module

cd /opt/module

mv apache-flume-1.11.0-bin.tar.gz flume1.11

2.配置Flume环境变量，编辑文件/etc/profile,添加内容，输入命令使其生效

vim /etc/profile

添加内容

export FLUME_HOME=/opt/module/flume1.11

export PATH=$PATH:$FLUME_HOME/bin

使配置生效

source /etc/profile

3.配置Flume配置文件

cd /opt/module/flume1.11/conf

cp flume-env.sh.template flume-env.sh

vim flume-env.sh

添加内容：

JAVA_HOME=/opt/module/jdk1.8

4.验证

flume-ng version

5.配置文件监控NameNode 日志文件

cp flume-conf.properties.template flume-conf.properties

vim flume-conf.properties

添加内容：

a1.sources = r1

a1.sinks = k1

a1.channels = c1

a1.sources.r1.type = exec

a1.sources.r1.command = tail -F /opt/module/hadoop3/logs/hadoop-root-namenode-master.log

a1.sinks.k1.type = hdfs

a1.sinks.k1.hdfs.path = hdfs://master:9000/tmp/flume/%Y%m%d（该路径根据要求可改）

a1.sinks.k1.hdfs.filePrefix = log-

a1.sinks.k1.hdfs.fileType = DataStream

a1.sinks.k1.hdfs.useLocalTimeStamp = true

a1.channels.c1.type = memory

a1.sources.r1.channels = c1

a1.sinks.k1.channel = c1

6.启动Flume传输Hadoop日志（ flume必须持有hadoop相关的包才能将数据输出到hdfs）

1)将如下包上传到flume/lib下 

cp $HADOOP_HOME/share/hadoop/common/hadoop-common-3.1.3.jar  /opt/module/flume1.11/lib

cp $HADOOP_HOME/share/hadoop/common/lib/hadoop-auth-3.1.3.jar /opt/module/flume1.11/lib

cp $HADOOP_HOME/share/hadoop/common/lib/commons-configuration2-2.1.1.jar /opt/module/flume1.11/lib

2)将hadoop的hdfs-site.xml和core-site.xml 放到flume/conf下

cp $HADOOP_HOME/etc/hadoop/core-site.xml /opt/module/flume1.11/conf

cp $HADOOP_HOME/etc/hadoop/hdfs-site.xml /opt/module/flume1.11/conf

3)flume1.11/conf目录下

rm /opt/module/flume1.11/lib/guava-11.0.2.jar

7.启动flume 

flume-ng agent --conf conf/ --conf-file /opt/module/flume1.11/conf/flume-conf.properties --name a1  -Dflume.root.logger=DEBUG,console