flume案例三：扇入

扇入(fan in)-Flume与Flume之间数据传递，多Flume汇总数据到单Flume

**目标 ：**agent1 监控.log文件 agent2 监控某一个端口的数据流
汇总数据 发送给agent3 并写道hdfs上

#agent1  监控.log文件
a1.sources=s1
a1.sinks=k1
a1.channels=c1

#设置sources
a1.sources.s1.type=spooldir
a1.sources.s1.spoolDir=/root/data/flumeFile/test
a1.sources.s1.includePattern=tmp_[0-9]{4}-[0-9]{2}-[0-9]{2}.log
a1.sources.s1.deserializer=LINE
a1.sources.s1.deserializer.maxLineLength=60000
#设置channel
a1.channels.c1.type=memory
a1.channels.c1.capacity=1000
a1.channels.c1.transactionCapacity=100
#设置sink
a1.sinks.k1.type=avro
a1.sinks.k1.hostname=localhost
a1.sinks.k1.port=5555
#设置连接
a1.sources.s1.channels=c1
a1.sinks.k1.channels=c1

#agent2  监控某一个端口的数据流
a2.sources= s1
a2.sinks =k1
a2.channels =c1
#设置sources
a2.sources.s1.type=netcat
a2.sources.s1.hostname=localhost
a2.sources.s1.port=1111
#设置channel
a2.channels.c1.type=memory
a2.channels.c1.capacity=1000
a2.channels.c1.transactionCapacity=100
#设置sinks
a2.sinks.k1.type=avro
a2.sinks.k1.hostname=localhost
a2.sinks.k1.port=5555
#设置连接
a2.sources.s1.channels=c1
a2.sinks.k1.channels=c1

#agent3 监听avro端口 汇总外部来的avro客户端的数据，并写道hdfs上
a3.sources= s1
a3.sinks =k1
a3.channels =c1
#设置sources
a3.sources.s1.type=avro
a3.sources.s1.hostname=localhost
a3.sources.s1.port=5555
#设置channel
a3.channels.c1.type=memory
a3.channels.c1.capacity=1000
a3.channels.c1.transactionCapacity=100
#设置sink
a3.sinks.k1.type=hdfs
a3.sinks.k1.hdfs.fileType=DataStream
a3.sinks.k1.hdfs.filePrefix=tmp
a3.sinks.k1.hdfs.fileSuffix=.csv
a3.sinks.k1.hdfs.path=hdfs://192.168.241.132:9000/flume/%Y-%m-%d
a3.sinks.k1.hdfs.useLocalTimeStamp=true
a3.sinks.k1.hdfs.batchSize=640
a3.sinks.k1.hdfs.rollCount= 0
a3.sinks.k1.hdfs.rollSize=6400000
a3.sinks.k1.hdfs.rollInterval =30
#设置连接
a3.sources.s1.channels=c1
a3.sinks.k1.channel=c1

ps:agent1 和agent2中的sink avro 端口号要保持一致 以及agent的source端口也要一样
启动顺序：a2和a1 随意 但要先启动a3

$ bin/flume-ng agent --conf conf/ --name a3 --conf-file jobconf/flume33.conf

$ bin/flume-ng agent --conf conf/ --name a2 --conf-file jobconf/flume22.conf

$ bin/flume-ng agent --conf conf/ --name a1 --conf-file jobconf/flume11.conf