flume操作

一、Flume安装配置
1、下载：http://www.apache.org/dyn/closer.lua/flume/1.9.0/apache-flume-1.9.0-bin.tar.gz
2、上传到/export/software：xftp
3、解压：tar -zxvf /export/software/apache-flume-1.9.0-bin.tar.gz -C /export/servers/
4、重命名
cd /export/servers/
mv apache-flume-1.9.0-bin/ flume
5、修改配置文件：
cd flume/conf/
cp flume-env.sh.template flume-env.sh
vi flume-env.sh
修改export JAVA_HOME=/export/servers/jdk
6、修改环境变量配置文件
vi /etc/profile
增加：export FLUME_HOME=/export/servers/flume
export PATH=$PATH:$FLUME_HOME/bin
后：source /etc/profile
7、在/export/data下创建目录
mkdir flumedata
在flumedata目录下创建文件：vi netcat-logger.conf,内容如下：
a1.sources=r1
a1.sinks=k1
a1.channels=c1
a1.sources.r1.type=netcat
a1.sources.r1.bind=localhost
a1.sources.r1.port=44444
a1.sinks.k1.type=logger
a1.channels.c1.type=memory
a1.channels.c1.capacity=1000
a1.channels.c1.transactionCapacity=1000
a1.sources.r1.channels=c1
a1.sinks.k1.channel=c1
8、使用指定采集方案启动flume
flume-ng agent --conf /export/data/flumedata/ --conf-file /export/data/flumedata/netcat-logger.conf --name a1 -Dflume.root.logger=INFO,console
9、复制会话，安装telnet
yum -y install telnet
10、telnet测试
telnet localhost 44444
可以查看原会话中flume状态
二、Flume的可靠性保证（负载均衡）
1、分发flume目录和profile配置文件
scp -r /export/servers/flume/ hadoop02:/export/servers/
scp -r /export/servers/flume/ hadoop03:/export/servers/
scp /etc/profile hadoop02:/etc/
scp /etc/profile hadoop03:/etc/
source /etc/profile
2、配置flume采集方案
在hadoop01上配置第一级采集配置
cd /export/servers/flume/conf/
cat exec-avro.conf 
内容如下：
a1.sources=r1
a1.sinks=k1 k2
a1.channels=c1
a1.sources.r1.channels=c1
a1.sources.r1.type=exec
a1.sources.r1.command=tail -F /root/logs/123.log
a1.channels.c1.type=memory
a1.channels.c1.capacity=1000
a1.channels.c1.transactionCapacity=100
a1.sinks.k1.channel=c1
a1.sinks.k1.type=avro
a1.sinks.k1.hostname=hadoop02
a1.sinks.k1.port=52020
a1.sinks.k2.channel=c1
a1.sinks.k2.type=avro
a1.sinks.k2.hostname=hadoop03
a1.sinks.k2.port=52020
a1.sinkgroups=g1
a1.sinkgroups.g1.sinks=k1 k2
a1.sinkgroups.g1.processor.type=load_balance
a1.sinkgroups.g1.processor.backoff=true
a1.sinkgroups.g1.processor.selector=random
a1.sinkgroups.g1.processor.maxTimeOut=10000
（2）在hadoop02上配置第二级采集方案
cd /export/servers/flume/conf/
cat avro-logger.conf 
内容如下：
a1.sources=r1
a1.sinks=k1
a1.channels=c1
a1.sources.r1.type=avro
a1.sources.r1.bind=hadoop02
a1.sources.r1.port=52020
a1.sinks.k1.type=logger
a1.channels.c1.type=memory
a1.channels.c1.capacity=1000
a1.channels.c1.transactionCapacity=100
a1.sources.r1.channels=c1
a1.sinks.k1.channel=c1
（3）在hadoop03上配置第二级采集方案
cd /export/servers/flume/conf/
cat avro-logger.conf 
内容如下：
a1.sources=r1
a1.sinks=k1
a1.channels=c1
a1.sources.r1.type=avro
a1.sources.r1.bind=hadoop03
a1.sources.r1.port=52020
a1.sinks.k1.type=logger
a1.channels.c1.type=memory
a1.channels.c1.capacity=1000
a1.channels.c1.transactionCapacity=100
a1.sources.r1.channels=c1
a1.sinks.k1.channel=c1
3、启动flume系统
（1）hadoop02的/export/servers/flume下执行：
flume-ng agent --conf conf/ --conf-file conf/avro-logger.conf --name a1 -Dflume.root.logger=INFO,console
（2）hadoop03的/export/servers/flume下执行：
flume-ng agent --conf conf/ --conf-file conf/avro-logger.conf --name a1 -Dflume.root.logger=INFO,console
(3)hadoop01上
在家目录下创建logs目录
执行：
flume-ng agent --conf conf/ --conf-file conf/exec-avro.conf --name a1 -Dflume.root.logger=INFO,console
4、flume系统负载均衡测试
hadoop01上克隆终端，执行：
while true;do echo "access access ..." >> /root/logs/123.log;sleep 1;done