Flume负载均衡与故障转移原理及实现

原理

负载均衡：将多个sink逻辑上分为一个sink组，sink组配合不同的SinkProcessor将数据相对均匀的分发到指定目录或者其他agent实例。
故障转移：有主备agent，主agent负责数据的采集、传输、落地，备用agent一直处于监听状态，一旦主agent宕机，备用agent启动，进行主agent的工作，直到主agent恢复。

案例

使用Flume1监控一个端口，其sink组中的sink分别对接Flume2和Flume3，采用FailoverSinkProcessor，实现故障转移的功能。

如图所示，flume1采集数据，根据SinkProcessor的类型分发数据到不同的Sink，如果SinkProcessor的类型指定为failover，就是故障转移实例；如果SinkProcessor的类型指定为load_balance，就是负载均衡的实例。

配置文件

# agent1
# 定义sources、channels、sinks
a1.sources = r1
a1.channels = c1
a1.sinks = k1 k2

# 配置source
a1.sources.r1.type = netcat
a1.sources.r1.bind = hadoop01
a1.sources.r1.port = 11111

# 配置channel
a1.channels.c1.type = memory

# 配置sink组
a1.sinkgroups = g1
a1.sinkgroups.g1.sinks = k1 k2
# 配置SinkProcessor：failover是故障转移，load_balance是负载均衡
# a1.sinkgroups.g1.processor.type = failover
a1.sinkgroups.g1.processor.type = load_balance
# 如果是负载均衡，还需要配置负载均衡策略 round_robin或者random
a1.sinkgroups.g1.processor.selector = round_robin

# 配置sink:k1发送数据到hadoop02
a1.sinks.k1.type = avro
a1.sinks.k1.hostname = hadoop02
a1.sinks.k1.port = 22222

# 配置sink k2:数据发送到hadoop03的33333端口
a1.sinks.k2.type = avro
a1.sinks.k2.hostname = hadoop03
a1.sinks.k2.port = 33333

# 配置source、channel、sink关系
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1
a1.sinks.k2.channel = c1

# agent2
# 定义sources、channels、sinks
a1.sources = r1
a1.channels = c1
a1.sinks = k1

# 配置source 从指定地址的端口接收数据
a1.sources.r1.type = avro
a1.sources.r1.bind = hadoop02
a1.sources.r1.port = 22222

# 配置channel
a1.channels.c1.type = memory

# 配置sink 
a1.sinks.k1.type = logger
# 测试logger打印到控制台


# 配置source、sink、channel关系
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1

# agent3
# 定义sources、channels、sinks
a1.sources = r1
a1.channels = c1
a1.sinks = k1

# 配置source 从指定地址的端口接收数据
a1.sources.r1.type = avro
a1.sources.r1.bind = hadoop03
a1.sources.r1.port = 33333

# 配置channel
a1.channels.c1.type = memory

# 配置sink 
a1.sinks.k1.type = logger
# 测试logger打印到控制台


# 配置source、sink、channel关系
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1