Flume 的应用5（负载均衡和故障转移及案例分析）

1、负载均衡和故障转移

• Flume负载均衡或故障转移

• Flume 支持使用将多个sink逻辑上分到一个sink组，sink 组配合不同的 SinkProcessor 可以实现负载均衡和错误恢复的功能。

2、负载均衡和故障转移案例

1）案例需求

• 使用 Flume1 监控一个端口，其 sink 组中的 sink 分别对接 Flume2 和 Flume3，采用 FailoverSinkProcessor，实现故障转移的功能。

2）需求分析

3）实现步骤

（1）准备工作

• 在/opt/module/flume-1.9.0/jobs目录下创建group2文件夹

[xiaoxq@hadoop105 jobs]$ mkdir group2
[xiaoxq@hadoop105 jobs]$ cd group2

（2）创建 flume-netcat-flume.conf

• 配置1个 netcat source 和 1个 channel、1个 sink group（2个sink），分别输送给 flume-flume-console1 和flume-flume-console2。
• 编辑配置文件

[xiaoxq@hadoop105 group2]$ vim flume-netcat-flume.conf

• 添加如下内容

# Name the components on this agent
a1.sources = r1
a1.channels = c1
a1.sinkgroups = g1
a1.sinks = k1 k2

# Describe/configure the source
a1.sources.r1.type = netcat
a1.sources.r1.bind = localhost
a1.sources.r1.port = 44444

a1.sinkgroups.g1.processor.type = failover
a1.sinkgroups.g1.processor.priority.k1 = 5
a1.sinkgroups.g1.processor.priority.k2 = 10
a1.sinkgroups.g1.processor.maxpenalty = 10000

# Describe the sink
a1.sinks.k1.type = avro
a1.sinks.k1.hostname = hadoop105
a1.sinks.k1.port = 4141

a1.sinks.k2.type = avro
a1.sinks.k2.hostname = hadoop105
a1.sinks.k2.port = 4142

# Describe the channel
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100

# Bind the source and sink to the channel
a1.sources.r1.channels = c1
a1.sinkgroups.g1.sinks = k1 k2
a1.sinks.k1.channel = c1
a1.sinks.k2.channel = c1

（3）创建flume-flume-console1.conf

• 配置上级 Flume 输出的 Source，输出是到本地控制台。
• 编辑配置文件

[xiaoxq@hadoop105 group2]$ vim flume-flume-console1.conf

• 添加如下内容

# Name the components on this agent
a2.sources = r1
a2.sinks = k1
a2.channels = c1

# Describe/configure the source
a2.sources.r1.type = avro
a2.sources.r1.bind = hadoop105
a2.sources.r1.port = 4141

# Describe the sink
a2.sinks.k1.type = logger

# Describe the channel
a2.channels.c1.type = memory
a2.channels.c1.capacity = 1000
a2.channels.c1.transactionCapacity = 100

# Bind the source and sink to the channel
a2.sources.r1.channels = c1
a2.sinks.k1.channel = c1

（4）创建 flume-flume-console2.conf

• 配置上级Flume输出的Source，输出是到本地控制台。
• 编辑配置文件

[xiaoxq@hadoop105 group2]$ vim flume-flume-console2.conf

• 添加如下内容

# Name the components on this agent
a3.sources = r1
a3.sinks = k1
a3.channels = c2

# Describe/configure the source
a3.sources.r1.type = avro
a3.sources.r1.bind = hadoop105
a3.sources.r1.port = 4142

# Describe the sink
a3.sinks.k1.type = logger

# Describe the channel
a3.channels.c2.type = memory
a3.channels.c2.capacity = 1000
a3.channels.c2.transactionCapacity = 100

# Bind the source and sink to the channel
a3.sources.r1.channels = c2
a3.sinks.k1.channel = c2

（5）执行配置文件

• 分别开启对应配置文件：flume-flume-console2，flume-flume-console1，flume-netcat-flume。

[xiaoxq@hadoop105 flume-1.9.0]$ bin/flume-ng agent -n a3 -c conf/ -f jobs/group2/flume-flume-console2.conf -Dflume.root.logger=INFO,console

[xiaoxq@hadoop105 flume-1.9.0]$ bin/flume-ng agent -n a2 -c conf/ -f jobs/group2/flume-flume-console1.conf -Dflume.root.logger=INFO,console

[xiaoxq@hadoop105 flume-1.9.0]$ bin/flume-ng agent -n a1 -c conf/ -f jobs/group2/flume-netcat-flume.conf -Dflume.root.logger=INFO,console

（6）使用 netcat 工具向本机的44444端口发送内容

[xiaoxq@hadoop105 group2]$ nc localhost 44444
hello
OK
java
OK
hadoop
OK
hive
OK
azkaban
OK
ahf
OK
diuafh
OK
asdhjf
OK
ab
OK
xiao
OK
xiang
OK
qiong


helloworld
OK
masql
OK

wei 
OK
dsjhf
OK
xiao
OK
xiang
OK

（7）查看 Flume2（console2） 及 Flume3 （console1）的控制台打印日志

（8）将Flume2（console2） kill，观察Flume3（console1）的控制台打印情况。

注：使用jps -ml查看Flume进程。

• 重新启动 console2 后