flume配置负载均衡

集群配置了3台机器，分别为master,slave,slave2

ip地址分别为192.168.88.128,192.168.88.129,192.168.88.130

其中master的配置为:

agent.sources = source1
agent.channels = channel1
agent.sinks = sink1 sink2
agent.sinkgroups = group1

agent.sinkgroups.group1.sinks = sink1 sink2
agent.sinkgroups.group1.processor.type = load_balance
agent.sinkgroups.group1.processor.selector = round_robin
agent.sinkgroups.group1.processor.backoff = true

agent.channels.channel1.type = file
agent.channels.channel1.checkpointDir = /tmp/flume/loadcheckpoint
agent.channels.channel1.dataDirs = /tmp/flume/loaddata

# For each one of the sources, the type is defined
agent.sources.source1.type = exec

agent.sources.source1.command = tail -F /home/bigdata/datasource/flume_user_action.txt

# The channel can be defined as follows.
agent.sources.source1.channels = channel1

agent.sinks.sink1.channel = channel1
agent.sinks.sink1.type = avro
agent.sinks.sink1.hostname = slave
agent.sinks.sink1.port = 41414
#Specify the channel the sink should use
agent.sinks.sink2.channel = channel1
agent.sinks.sink2.type = avro
agent.sinks.sink2.hostname = slave2
agent.sinks.sink2.port = 41414

slave的配置为:

agent.sources = source1
agent.channels = channel1
agent.sinks = sink1

agent.sources.source1.type = avro
agent.sources.source1.bind = 192.168.88.129
agent.sources.source1.port = 41414
agent.sources.source1.channels= channel1

agent.channels.channel1.type = file
agent.channels.channel1.checkpointDir = /home/bigdata/tmp/flume/checkpoint
agent.channels.channel1.dataDirs = /home/bigdata/tmp/flume/channeldata

agent.sinks.sink1.channel = channel1
agent.sinks.sink1.type = file_roll
agent.sinks.sink1.sink.directory = /home/bigdata/tmp/flume/loaddata
agent.sinks.sink1.sink.rollInterval = 0

slave2的配置为:

agent.sources = source1
agent.channels = channel1
agent.sinks = sink1


agent.sources.source1.type = avro
agent.sources.source1.bind = 192.168.88.130
agent.sources.source1.port = 41414
agent.sources.source1.channels = channel1


agent.channels.channel1.type = file
agent.channels.channel1.checkpointDir = /home/bigdata/tmp/flume/checkpoint
agent.channels.channel1.dataDirs = /home/bigdata/tmp/flume/channeldata


agent.sinks.sink1.channel = channel1
agent.sinks.sink1.type = file_roll
agent.sinks.sink1.sink.directory = /home/bigdata/tmp/flume/loaddata
agent.sinks.sink1.sink.rollInterval = 0

有几个地方要注意一下，第一个是agent.sources.source1.bind的地址必须填ip地址，不然master的sink写不过来

报错如下：

Exception in thread "main" org.apache.flume.FlumeException:NettyAvroRpcClient{

host:xxxx,port:xxxx}:RPC connection error

第二个是agent.sinks.sink1.sink.directory这个参数的目录必须存在，不然写不进去

报错如下：