Flume-负载均衡和故障转移案例

负载均衡和故障转移

使用Flume监控一个端口，其sink组中的sink分别对接Flume2和Flume3，采用FailoverSinkProcessor，实现故障转移的功能。其中Flume2和Flume3的Sink直接采用Logger方便查看和测试。

FailoverSinkProcessor：故障转移处理器维护了一个发送Event失败的sink的列表，保证有一个sink是可用的来发送Event。

配置及说明如下：

故障转移机制的工作原理是将故障sink降级到一个池中，在池中为它们分配冷却期（超时时间），默认30s。 在到达冷却时间之后，它将恢复到实时池。在冷却期中的sink不被考虑使用，只有恢复到实时池中的才能通过优先级被考虑使用。

sink具有与之相关的优先级，数值越大，优先级越高。 如果在发送Event时Sink发生故障，会继续尝试下一个具有最高优先级的sink。 例如，有100、80、50三个优先级的sink，优先级100的sink发生了故障则使用优先级有80的sink。如果未指定优先级，则根据配置中的顺序来选取。

要使用故障转移选择器，不仅要设置sink组的选择器为failover，还有为每一个sink设置一个唯一的优先级数值。 可以使用 maxpenalty 属性设置故障转移时间的上限（毫秒）。

Flume1配置文件如下：

# Name the components on this agent
a1.sources = r1
a1.sinks = k1 k2
a1.channels = c1

# Describe/configure the source
a1.sources.r1.type = netcat
a1.sources.r1.bind = localhost
a1.sources.r1.port = 44444

# Describe the sink
a1.sinks.k1.type = avro
a1.sinks.k1.hostname = hadoop113
a1.sinks.k1.port = 4441
a1.sinks.k2.type = avro
a1.sinks.k2.hostname = hadoop113
a1.sinks.k2.port = 4442

# Sink Group
# channel对多个sink，默认是负载均衡，且是轮询的
a1.sinkgroups = g1
a1.sinkgroups.g1.sinks = k1 k2
a1.sinkgroups.g1.processor.type = failover
a1.sinkgroups.g1.processor.priority.k1 = 5
a1.sinkgroups.g1.processor.priority.k2 = 10
a1.sinkgroups.g1.processor.maxpenalty = 10000

# Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100

# Bind the source and sink to the channel
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1
a1.sinks.k2.channel = c1

Flume2配置文件如下：

# Name the components on this agent
a2.sources = r2
a2.sinks = k2
a2.channels = c2

# Describe/configure the source
a2.sources.r2.type = avro
a2.sources.r2.bind = hadoop113
a2.sources.r2.port = 4441

# Describe the sink
a2.sinks.k2.type = logger

# Use a channel which buffers events in memory
a2.channels.c2.type = memory
a2.channels.c2.capacity = 1000
a2.channels.c2.transactionCapacity = 100

# Bind the source and sink to the channel
a2.sources.r2.channels = c2
a2.sinks.k2.channel = c2

Flume3配置文件如下：

# Name the components on this agent
a3.sources = r2
a3.sinks = k2
a3.channels = c2

# Describe/configure the source
a3.sources.r2.type = avro
a3.sources.r2.bind = hadoop113
a3.sources.r2.port = 4442

# Sink
a3.sinks.k2.type = logger


# Use a channel which buffers events in memory
a3.channels.c2.type = memory
a3.channels.c2.capacity = 1000
a3.channels.c2.transactionCapacity = 100

# Bind the source and sink to the channel
a3.sources.r2.channels = c2
a3.sinks.k2.channel = c2

通过nc localhost 44444连接到44444端口，依次输入hello、word、hahahah，结果全部在Flume3中输出了。

此时我们关闭Flume3，依次输入test、failover，结果全在Flume中输出了。

说明故障转移成功了。

负载均衡的话只需要将Sink组的SinkProcessor设置成load_balance即可，其他的和故障转移一模一样

# Sink Group
# channel对多个sink，不设置的话默认是负载均衡，且是轮询的
a1.sinkgroups = g1
a1.sinkgroups.g1.sinks = k1 k2
a1.sinkgroups.g1.processor.type = load_balance
a1.sinkgroups.g1.processor.backoff = true
## 这里设置成random
a1.sinkgroups.g1.processor.selector = random

运行结果如下：