flume复制及多路复用案例

复制和多路复用

Flume支持将事件流向一个或者多个目的地。这种模式可以将相同数据复制到多个channel中，或者将不同数据分发到不同的channel中，sink可以选择传送到不同的目的地。

案例：

整体flume架构如上图所示，下面开始编辑flume配置文件：

在hadoop103节点的 /opt/module/flume/job 下新建配置文件：

vim netcat-flume-loggers.conf

添加如下内容：

a3.sources = r3
a3.channels = c3
a3.sources.r3.type = avro
a3.sources.r3.channels = c3
a3.sources.r3.bind = hadoop103
a3.sources.r3.port = 4141

a3.channels.c3.type = memory
a3.channels.c3.capacity = 1000
a3.channels.c3.transactionCapacity = 100

a3.sinks = k3
a3.sinks.k3.type = logger
a3.sinks.k3.channel = c3

在hadoop104节点的 /opt/module/flume/job 下新建配置文件：

vim netcat-flume-loggers.conf

添加如下内容：

a4.sources = r4
a4.channels = c4
a4.sources.r4.type = avro
a4.sources.r4.channels = c4
a4.sources.r4.bind = hadoop104
a4.sources.r4.port = 4141

a4.channels.c4.type = memory
a4.channels.c4.capacity = 1000
a4.channels.c4.transactionCapacity = 100

a4.sinks = k4
a4.sinks.k4.type = logger
a4.sinks.k4.channel = c4

在hadoop102节点的 /opt/module/flume/job 下新建配置文件：

vim netcat-flume-loggers.conf

添加如下内容：

a2.sources = r1
a2.sinks = k1 k2
a2.channels = c1 c2

# Describe/configure the source
a2.sources.r1.type = netcat
a2.sources.r1.bind = hadoop102
a2.sources.r1.port = 44444

# Describe the sink
a2.sinks.k1.type = avro
a2.sinks.k1.hostname = hadoop103
a2.sinks.k1.port = 4141

a2.sinks.k2.type = avro
a2.sinks.k2.hostname = hadoop104
a2.sinks.k2.port = 4141

# Use a channel which buffers events in memory
a2.channels.c1.type = memory
a2.channels.c1.capacity = 1000
a2.channels.c1.transactionCapacity = 100

# Use a channel which buffers events in memory
a2.channels.c2.type = memory
a2.channels.c2.capacity = 1000
a2.channels.c2.transactionCapacity = 100

# Bind the source and sink to the channel
a2.sources.r1.channels = c1 c2
a2.sinks.k1.channel = c1
a2.sinks.k2.channel = c2

三台服务器的配置文件建好了，现在就可以启动flume集群了：

先启动hadoop103和hadoop104节点的logger服务端：

[atguigu@hadoop104 flume]$ bin/flume-ng agent -n a4 -c conf/ -f job/netcat-flume-loggers.conf -Dflume.root.logger=INFO,console

[atguigu@hadoop103 flume]$ bin/flume-ng agent -n a3 -c conf/ -f job/netcat-flume-loggers.conf -Dflume.root.logger=INFO,console 

启动hadoop102节点的netcat：

[atguigu@hadoop102 flume]$ bin/flume-ng agent -n a2 -c conf/ -f job/netcat-flume-loggers.conf

开启netcat后此窗口就不能操作了，再新建一个hadoop102窗口启动nc：

[atguigu@hadoop102 ~]$ nc hadoop102 44444

job/netcat-flume-loggers.conf

开启netcat后此窗口就不能操作了，再新建一个hadoop102窗口启动nc：

[atguigu@hadoop102 ~]$ nc hadoop102 44444

此时在nc窗口下输入消息，hadoop103和hadoop104均可收到。