案例三、Flume 与 Flume 之间数据传递,多 Flume 汇总数据到单 Flume。
目标:flume-fanin-1监控某一个端口的数据流,flume-fanin-2 监控文件,flume-fanin-1 和 flume-fanin-2 将数据发送给 flume-fanin-3,flume-fanin-3 将最终数据写入到HDFS。
分步实现
1.创建 flume-fanin-1.conf,用于监控端口 55555,同时 sink 数据到 flume-fanin-3
# 1 agent
a1.sources = netcat-a1
a1.sinks = avro113
a1.channels = c1
# 2 source
a1.sources.netcat-a1.type = netcat
a1.sources.netcat-a1.bind = bigdata111
a1.sources.netcat-a1.port = 55555
#3 sink
a1.sinks.avro113.type = avro
a1.sinks.avro113.hostname = bigdata113
a1.sinks.avro113.port = 4141
# 4 channel
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100
# 5 Bind
a1.sources.netcat-a1.channels = c1