复制和多路复用
Flume支持将事件流向一个或者多个目的地。这种模式可以将相同数据复制到多个channel中,或者将不同数据分发到不同的channel中,sink可以选择传送到不同的目的地。
案例:
整体flume架构如上图所示,下面开始编辑flume配置文件:
在hadoop103节点的 /opt/module/flume/job 下新建配置文件:
vim netcat-flume-loggers.conf
添加如下内容:
a3.sources = r3
a3.channels = c3
a3.sources.r3.type = avro
a3.sources.r3.channels = c3
a3.sources.r3.bind = hadoop103
a3.sources.r3.port = 4141
a3.channels.c3.type = memory
a3.channels.c3.capacity = 1000
a3.channels.c3.transactionCapacity = 100
a3.sinks = k3
a3.sinks.k3.type = logger
a3.sinks.k3.channel = c3
在hadoop104节点的 /opt/module/flume/job 下新建配置文件:
vim netcat-flume-loggers.conf
添加如下内容:
a4.sources = r4
a4.channels = c4
a4.sources.r4.type = avro
a4.sources.r4.channels = c4
a4.sources.r4.bind = hadoop104
a4.sources.r4.port = 4141
a4.channels.c4.type = memory
a4.channels.c4.capacity = 1000
a4.channels.c4.transactionCapacity = 100
a4.sinks = k4
a4.sinks.k4.type = logger
a4.sinks.k4.channel = c4
在hadoop102节点的 /opt/module/flume/job 下新建配置文件:
vim netcat-flume-loggers.conf
添加如下内容:
a2.sources = r1
a2.sinks = k1 k2
a2.channels = c1 c2
# Describe/configure the source
a2.sources.r1.type = netcat
a2.sources.r1.bind = hadoop102
a2.sources.r1.port = 44444
# Describe the sink
a2.sinks.k1.type = avro
a2.sinks.k1.hostname = hadoop103
a2.sinks.k1.port = 4141
a2.sinks.k2.type = avro
a2.sinks.k2.hostname = hadoop104
a2.sinks.k2.port = 4141
# Use a channel which buffers events in memory
a2.channels.c1.type = memory
a2.channels.c1.capacity = 1000
a2.channels.c1.transactionCapacity = 100
# Use a channel which buffers events in memory
a2.channels.c2.type = memory
a2.channels.c2.capacity = 1000
a2.channels.c2.transactionCapacity = 100
# Bind the source and sink to the channel
a2.sources.r1.channels = c1 c2
a2.sinks.k1.channel = c1
a2.sinks.k2.channel = c2
三台服务器的配置文件建好了,现在就可以启动flume集群了:
先启动hadoop103和hadoop104节点的logger服务端:
[atguigu@hadoop104 flume]$ bin/flume-ng agent -n a4 -c conf/ -f job/netcat-flume-loggers.conf -Dflume.root.logger=INFO,console
[atguigu@hadoop103 flume]$ bin/flume-ng agent -n a3 -c conf/ -f job/netcat-flume-loggers.conf -Dflume.root.logger=INFO,console
启动hadoop102节点的netcat:
[atguigu@hadoop102 flume]$ bin/flume-ng agent -n a2 -c conf/ -f job/netcat-flume-loggers.conf
开启netcat后此窗口就不能操作了,再新建一个hadoop102窗口启动nc:
[atguigu@hadoop102 ~]$ nc hadoop102 44444
job/netcat-flume-loggers.conf
开启netcat后此窗口就不能操作了,再新建一个hadoop102窗口启动nc:
[atguigu@hadoop102 ~]$ nc hadoop102 44444
此时在nc窗口下输入消息,hadoop103和hadoop104均可收到。