flume配置负载均衡、故障转移

flume配置负载均衡、故障转移

---

1.flume负载均衡

第一台机器配置

#负载均衡
#配置Agent a1的组件
a1.sources=r1
a1.channels=c1
a1.sinks=s1 s2

#配置a1的source
a1.sources.r1.type=netcat
a1.sources.r1.bind=0.0.0.0
a1.sources.r1.port=3333

##配置a1的channel
a1.channels.c1.type=memory
a1.channels.c1.capacity=1000
a1.channels.c1.trancactionCapacity=100

#配置a1的sink
a1.sinks.s1.type=avro
a1.sinks.s1.hostname=node2
a1.sinks.s1.port=8888

a1.sinks.s2.type=avro
a1.sinks.s2.hostname=node3
a1.sinks.s2.port=8888

#配置sink组以及sink处理器运行
a1.sinkgroups = g1
a1.sinkgroups.g1.sinks = s1 s2
a1.sinkgroups.g1.processor.type = load_balance
a1.sinkgroups.g1.processor.backoff = true
#参数可选round_robin（轮询）或者random（随机）
a1.sinkgroups.g1.processor.selector = round_robin

#绑定
a1.sources.r1.channels=c1
a1.sinks.s1.channel=c1
a1.sinks.s2.channel=c1

第二台和第三台的配置一样：

#接收消息
 
# 配置Agent a1的组件
a1.sources=r1
a1.channels=c1
a1.sinks=s1
# 配置a1的source
a1.sources.r1.type=avro
a1.sources.r1.bind=0.0.0.0
a1.sources.r1.port=8888
# 配置a1的channel
a1.channels.c1.type=memory
# 配置a1的sink
a1.sinks.s1.type=logger
# 绑定
a1.sources.r1.channels=c1
a1.sinks.s1.channel=c1

2. flume故障转移

第一台机器配置：

#故障转移
#配置Agent a1的组件
a1.sources=r1
a1.channels=c1 c2
a1.sinks=s1 s2
#配置a1的source
a1.sources.r1.type=netcat
a1.sources.r1.bind=0.0.0.0
a1.sources.r1.port=3333
#配置a1的channel
a1.channels.c1.type=memory
a1.channels.c1.capacity=1000
a1.channels.c1.trancactionCapacity=100

a1.channels.c2.type=memory
a1.channels.c2.capacity=1000
a1.channels.c2.trancactionCapacity=100

#配置a1的sink
a1.sinks.s1.type=avro
a1.sinks.s1.hostname=node2
a1.sinks.s1.port=8888

a1.sinks.s2.type=avro
a1.sinks.s2.hostname=node3
a1.sinks.s2.port=8888

#配置sink组以及sink处理器运行
a1.sinkgroups = g1
a1.sinkgroups.g1.sinks = s1 s2
a1.sinkgroups.g1.processor.type = failover
a1.sinkgroups.g1.processor. priority.k1 = 5
a1.sinkgroups.g1.processor.priority.k2 = 10
a1.sinkgroups.g1.processor.maxpenalty = 1000

#绑定
a1.sources.r1.channels=c1 c2
a1.sinks.s1.channel=c1
a1.sinks.s2.channel=c2

第二台和第三台的配置一样：

#接收消息
 
# 配置Agent a1的组件
a1.sources=r1
a1.channels=c1
a1.sinks=s1
# 配置a1的source
a1.sources.r1.type=avro
a1.sources.r1.bind=0.0.0.0
a1.sources.r1.port=8888
# 配置a1的channel
a1.channels.c1.type=memory
# 配置a1的sink
a1.sinks.s1.type=logger
# 绑定
a1.sources.r1.channels=c1
a1.sinks.s1.channel=c1