flume实例（三）：扇入

1.实现功能：

扇入：flume可以实现从数以百计的Web servers中收集信息，然后将日志信息传送到十几个agent服务器，最后写到hdfs上。本文实例采用的就是实现这个扇入功能：

flume1：收集4666端口信息，并输送到Avro Sink

flume2：监控日志信息，并输送到Avro Sink

flume3：收集flume1 Avro Sink和flume2 Avro Sink的数据，写入到hdfs上。

 

2.代理

（1）flume1

# Name the components on this agent
a1.sources = r1
a1.sinks = k1
a1.channels = c1

# Describe/configure the source
a1.sources.r1.type = netcat
a1.sources.r1.bind = bigdata.ibeifeng.com
a1.sources.r1.port = 4666

# Describe the sink
a1.sinks.k1.type = avro
a1.sinks.k1.hostname = bigdata.ibeifeng.com
a1.sinks.k1.port = 6666

# Use a channel which buffers events in memory
a1.channels.c1.type = memory
#channe最多存储1000事件
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100

# Bind the source and sink to the channel
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1

（2）flume2

a2.sources = r2
a2.sinks = k2
a2.channels = c2

#定义source的类型
a2.sources.r2.type = exec
a2.sources.r2.command = tail -f  /var/log/httpd/access_log
a2.sources.r2.channels = c2

#定义channel
a2.channels.c2.type = memory
#存储1000个event事件
a2.channels.c2.capacity = 1000
a2.channels.c2.transactionCapacity = 100

          
#定义sink
a2.sinks.k2.type = avro
a2.sinks.k2.hostname = bigdata.ibeifeng.com
a2.sinks.k2.port = 6666
a2.sinks.k2.channel = c2

（3）flume3

a3.sources = r3
a3.sinks = k3
a3.channels = c3

#定义source的类型
a3.sources.r3.type = avro
a3.sources.r3.bind = bigdata.ibeifeng.com
a3.sources.r3.port = 6666
a3.sources.r3.channels = c3

#定义channel
a3.channels.c3.type = memory
#存储1000个event事件
a3.channels.c3.capacity = 1000
a3.channels.c3.transactionCapacity = 100

#定义sink
a3.sinks.k3.type = hdfs
a3.sinks.k3.hdfs.path = hdfs://bigdata.ibeifeng.com:8020/opo
a3.sinks.k3.channel = c3

 

3.启动

bin/flume-ng agent --name a3 --conf  conf --conf-file conf/avro-hdfs.properties &
bin/flume-ng agent --name a2 --conf  conf --conf-file conf/apache-avro.properties &
bin/flume-ng agent --name a1 --conf  conf --conf-file conf/flume-test2.properties &

（经测试，成功！）