高可用集群之~ 四：flume NG

本博客配置五个节点的 flume 集群
01,02,03 agents
04,05 collector

打开官网http://archive.cloudera.com/cdh5/cdh/5/

进入conf目录

配置flume-env.sh

添加 JAVA_HOME
export JAVA_HOME=/opt/bigdata/jdk18
##配置JAVA_OPTS
export JAVA_OPTS="-Xms100m -Xmx10000m -Dcom.sun.management.jmxremote"

添加环境变量

[root@cai conf]# vi /etc/profile.d/env.sh

export FLUME_HOME=/opt/bigdata/flume
export PATH=$FLUME_HOME/bin:$PATH

刷新环境变量

[root@cai conf]# source /etc/profile

利用 rsync 脚本分发 /etc/profile.d/env.sh 到各节点，要刷新
这是一个链接：点击此处：关于如何编写脚本和运行请参考 脚本专栏的xrsync 增量脚本

agent 配置如下(01,02,03配置一样)

flume-conf.properties

#agent1 name
agent1.channels = c1
agent1.sources = r1
agent1.sinks = k1 k2
#set gruop
agent1.sinkgroups = g1
#set channel
agent1.channels.c1.type = memory
agent1.channels.c1.capacity = 1000
agent1.channels.c1.transactionCapacity = 100
agent1.sources.r1.channels = c1
agent1.sources.r1.type = exec
agent1.sources.r1.command = tail -F /home/hadoop/flumetest/dir/logdfs/flumetest.log
agent1.sources.r1.interceptors = i1 i2
agent1.sources.r1.interceptors.i1.type = static
agent1.sources.r1.interceptors.i1.key = Type
agent1.sources.r1.interceptors.i1.value = LOGIN
agent1.sources.r1.interceptors.i2.type = timestamp
# set sink1
agent1.sinks.k1.channel = c1
agent1.sinks.k1.type = avro
agent1.sinks.k1.hostname = cluster04
agent1.sinks.k1.port = 52020
# set sink2
agent1.sinks.k2.channel = c1
agent1.sinks.k2.type = avro
agent1.sinks.k2.hostname = cluster05
agent1.sinks.k2.port = 52020
#set sink group
agent1.sinkgroups.g1.sinks = k1 k2
#set failover
agent1.sinkgroups.g1.processor.type = failover
agent1.sinkgroups.g1.processor.priority.k1 = 10
agent1.sinkgroups.g1.processor.priority.k2 = 1
agent1.sinkgroups.g1.processor.maxpenalty = 10000
~    

Collector1 (04)配置如下

flume-conf.properties

#set Agent name
a1.sources = r1
a1.channels = c1
a1.sinks = k1
#set channel
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100
# other node,nna to nns
a1.sources.r1.type = avro
a1.sources.r1.bind = cluster04
a1.sources.r1.port = 52020
a1.sources.r1.interceptors = i1
a1.sources.r1.interceptors.i1.type = static
a1.sources.r1.interceptors.i1.key = Collector
a1.sources.r1.interceptors.i1.value = cluster04
a1.sources.r1.channels = c1
#set sink to hdfs
a1.sinks.k1.type=hdfs
a1.sinks.k1.hdfs.path=hdfs://mycluster/flume/logdfs
a1.sinks.k1.hdfs.fileType=DataStream
a1.sinks.k1.hdfs.writeFormat=TEXT
a1.sinks.k1.hdfs.rollInterval=1
a1.sinks.k1.channel=c1
a1.sinks.k1.hdfs.filePrefix=%Y-%m-%d
a1.sinks.k1.hdfs.fileSuffix=.txt　　                                    

Collector2 (05)配置如下

flume-conf.properties

#set Agent name
a1.sources = r1
a1.channels = c1
a1.sinks = k1
#set channel
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100
# other node,nna to nns
a1.sources.r1.type = avro
a1.sources.r1.bind = cluster05
a1.sources.r1.port = 52020
a1.sources.r1.interceptors = i1
a1.sources.r1.interceptors.i1.type = static
a1.sources.r1.interceptors.i1.key = Collector
a1.sources.r1.interceptors.i1.value = cluster05
a1.sources.r1.channels = c1
#set sink to hdfs
a1.sinks.k1.type=hdfs
a1.sinks.k1.hdfs.path=hdfs://mycluster/flume/logdfs
a1.sinks.k1.hdfs.fileType=DataStream
a1.sinks.k1.hdfs.writeFormat=TEXT
a1.sinks.k1.hdfs.rollInterval=1
a1.sinks.k1.channel=c1
a1.sinks.k1.hdfs.filePrefix=%Y-%m-%d
a1.sinks.k1.hdfs.fileSuffix=.txt                                       

flume集群启动

在Collector各节点上启动命令如下所示：

./bin/flume-ng agent --conf conf/ --conf-file conf/flume-conf.properties --name a1 -Dflume.root.logger=INFO,console

注：命令中的a1表示配置文件中的Agent的Name，如配置文件中的a1。flume-server.properties表示配置文件所在配置，需填写准确的配置文件路径。
在Agent各节点上启动命令如下所示：

./bin/flume-ng agent --conf conf/ --conf-file conf/flume-conf.properties --name agent1 -Dflume.root.logger=INFO,console

测试

往被监测的文件中输入内容，查看 hdfs 是否写入文件