flume负载均衡版配置

flume负载均衡的意思：由node01 发送数据给node02 和node03 ,由他们俩分别轮询接收消息

设置node01,在flume安装包的conf目录下新建load_balancer.conf

#命名
a1.sources = r1
a1.channels = c1
a1.sinks = k1 k2

#设置资源
a1.sources.r1.type = exec
a1.sources.r1.command = tail -F /export/servers/taillogs/access_log

#设置通道channels
a1.channels.c1.type = memory
a1.channels.c1.capatity = 1000
a1.channels.c1.transactionCapacity = 100

#设置sink1
a1.sinks.k1.type=avro
a1.sinks.k1.hostname=node02
a1.sinks.k1.port=52020

#设置sink2
a1.sinks.k2.type=avro
a1.sinks.k2.hostname=node03
a1.sinks.k2.port=52020

#关系
a1.sources.r1.channels=c1
a1.sinks.k1.channel=c1
a1.sinks.k2.channel=c1

#设置proceecing
a1.sinkgroups=g1
a1.sinkgroups.g1.sinks=k1 k2
a1.sinkgroups.g1.processor.type=load_balance
a1.sinkgroups.g1.processor.backoff=true
a1.sinkgroups.g1.processor.selector=round_robin
a1.sinkgroups.g1.processor.selector.maxTimeOut=1000

 

设置node02 ,新建load_banlancer_server.conf

#命名
a1.sources = r1
a1.sinks = k1
a1.channels = c1

#资源
a1.sources.r1.type = avro
a1.sources.r1.bind = node02
a1.sources.r1.port = 52020

# sink
a1.sinks.k1.type = logger

#通道
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100

# 连接
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1

 

设置node02 ,新建load_banlancer_server.conf

#命名
a1.sources = r1
a1.sinks = k1
a1.channels = c1

#资源
a1.sources.r1.type = avro
a1.sources.r1.bind = node03
a1.sources.r1.port = 52020

# sink
a1.sinks.k1.type = logger

#通道
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100

# 连接
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1

 

分别开启node02 node03 和node01

bin/flume-ng agent -n a1 -c conf -f conf/load_banlancer_server.conf -Dflume.root.logger=DEBUG,console

 

往node01的/export/servers/taillogs/access_log追加数据看node02 和node03的变化

结果如下，数字都是字节流