flume负载均衡案例

负载均衡和故障转移

Flume支持使用将多个sink逻辑上分到一个sink组，sink组配合不同的SinkProcessor可以实现负载均衡和错误恢复的功能。

负载均衡案例：

整体flume架构如上图所示，下面开始编辑flume配置文件：

在hadoop103节点的 /opt/module/flume/job 下新建配置文件：

vim netcat-flume-loggers.conf

添加如下内容：

a3.sources = r3
a3.channels = c3
a3.sources.r3.type = avro
a3.sources.r3.channels = c3
a3.sources.r3.bind = hadoop103
a3.sources.r3.port = 4141

a3.channels.c3.type = memory
a3.channels.c3.capacity = 1000
a3.channels.c3.transactionCapacity = 100

a3.sinks = k3
a3.sinks.k3.type = logger
a3.sinks.k3.channel = c3

在hadoop104节点的 /opt/module/flume/job 下新建配置文件：

vim netcat-flume-loggers.conf

添加如下内容：

a4.sources = r4
a4.channels = c4
a4.sources.r4.type = avro
a4.sources.r4.channels = c4
a4.sources.r4.bind = hadoop104
a4.sources.r4.port = 4141

a4.channels.c4.type = memory
a4.channels.c4.capacity = 1000
a4.channels.c4.transactionCapacity = 100

a4.sinks = k4
a4.sinks.k4.type = logger
a4.sinks.k4.channel = c4

在hadoop102节点的 /opt/module/flume/job 下新建配置文件：

vim netcat-flume-loggers1.conf

添加如下内容：

a2.sources = r1
a2.sinks = k1 k2
a2.channels = c1

# Describe/configure the source
a2.sources.r1.type = netcat
a2.sources.r1.bind = hadoop102
a2.sources.r1.port = 44444

a2.sinkgroups = g1
a2.sinkgroups.g1.sinks = k1 k2
a2.sinkgroups.g1.processor.type = load_balance
#这里的true表示开启退避算法
a2.sinkgroups.g1.processor.backoff = true
a2.sinkgroups.g1.processor.selector = random

# Describe the sink
a2.sinks.k1.type = avro
a2.sinks.k1.hostname = hadoop103
a2.sinks.k1.port = 4141

a2.sinks.k2.type = avro
a2.sinks.k2.hostname = hadoop104
a2.sinks.k2.port = 4141

# Use a channel which buffers events in memory
a2.channels.c1.type = memory
a2.channels.c1.capacity = 1000
a2.channels.c1.transactionCapacity = 100


# Bind the source and sink to the channel
a2.sources.r1.channels = c1
a2.sinks.k1.channel = c1
a2.sinks.k2.channel = c1

先启动hadoop103和hadoop104节点的logger服务端：

[atguigu@hadoop104 flume]$ bin/flume-ng agent -n a4 -c conf/ -f job/netcat-flume-loggers.conf -Dflume.root.logger=INFO,console

[atguigu@hadoop103 flume]$ bin/flume-ng agent -n a3 -c conf/ -f job/netcat-flume-loggers.conf -Dflume.root.logger=INFO,console 

启动hadoop102节点的netcat：

[atguigu@hadoop102 flume]$ bin/flume-ng agent -n a2 -c conf/ -f job/netcat-flume-loggers1.conf

开启netcat后此窗口就不能操作了，再新建一个hadoop102窗口启动nc：

[atguigu@hadoop102 ~]$ nc hadoop102 44444

此时在nc窗口下输入消息，hadoop103和hadoop104均可收到。

此时hadoop103和hadoop104随机收取hadoop102发送的消息，若将hadoop102的配置文件中修改

a2.sinkgroups.g1.processor.selector = round_robin

p103和hadoop104均可收到。

此时hadoop103和hadoop104随机收取hadoop102发送的消息，若将hadoop102的配置文件中修改

a2.sinkgroups.g1.processor.selector = round_robin

那么hadoop103和hadoop104将以轮询的方式接收消息（在高速大量的数据下才能体现出来）。

故障转移案例：

hadoop103和hadoop104配置文件同上，在hadoop102的/opt/module/flume/job下新建一个配置文件：

[atguigu@hadoop102 job]$ vim netcat-flume-loggers2.conf

添加内容如下：

a2.sources = r1
a2.sinks = k1 k2
a2.channels = c1

# Describe/configure the source
a2.sources.r1.type = netcat
a2.sources.r1.bind = hadoop102
a2.sources.r1.port = 44444

a2.sinkgroups = g1
a2.sinkgroups.g1.sinks = k1 k2
a2.sinkgroups.g1.processor.type = failover
#这里的priority表示优先级，数值最大的sink将为active状态，其他的sink将为standby状态
#当active状态的sink异常时，standby状态的优先级最大的sink接替工作
a2.sinkgroups.g1.processor.priority.k1 = 5
a2.sinkgroups.g1.processor.priority.k2 = 10
a2.sinkgroups.g1.processor.maxpenalty = 10000

# Describe the sink
a2.sinks.k1.type = avro
a2.sinks.k1.hostname = hadoop103
a2.sinks.k1.port = 4141

a2.sinks.k2.type = avro
a2.sinks.k2.hostname = hadoop104
a2.sinks.k2.port = 4141

# Use a channel which buffers events in memory
a2.channels.c1.type = memory
a2.channels.c1.capacity = 1000
a2.channels.c1.transactionCapacity = 100


# Bind the source and sink to the channel
a2.sources.r1.channels = c1
a2.sinks.k1.channel = c1
a2.sinks.k2.channel = c1

先启动hadoop103和hadoop104节点的logger服务端：

[atguigu@hadoop104 flume]$ bin/flume-ng agent -n a4 -c conf/ -f job/netcat-flume-loggers.conf -Dflume.root.logger=INFO,console

[atguigu@hadoop103 flume]$ bin/flume-ng agent -n a3 -c conf/ -f job/netcat-flume-loggers.conf -Dflume.root.logger=INFO,console 

启动hadoop102节点的netcat：

[atguigu@hadoop102 flume]$ bin/flume-ng agent -n a2 -c conf/ -f job/netcat-flume-loggers2.conf

开启netcat后此窗口就不能操作了，再新建一个hadoop102窗口启动nc：

[atguigu@hadoop102 ~]$ nc hadoop102 44444

此时在nc窗口下输入消息，hadoop104可收到hadoop102发送的消息，若hadoop104异常，hadoop103替代hadoop104接收消息。