Flume负载均衡配置

flume负载均衡配置

 

 

集群DNS配置如下：

hadoop-maser 192.168.177.162

machine-0192.168.177.158

machine-1191.168.177.167

配置主Flume，在hadoop-maser机上。配置文件为loadbalance.properties。

agent.sources=s1

agent.channels=c1

agent.sinks=k1 k2

 agent.sinkgroups = g1

agent.sinkgroups.g1.sinks = k1 k2

agent.sinkgroups.g1.processor.type =load_balance

agent.sinkgroups.g1.processor.selector =round_robin

agent.sinkgroups.g1.processor.backoff =true

 

agent.sources.s1.type=avro

agent.sources.s1.channels=c1

agent.sources.s1.bind=0.0.0.0

agent.sources.s1.port=51515

agent.sources.s1.interceptors=i1

agent.sources.s1.interceptors.i1.type=timestamp

 

agent.channels.c1.type=memory

 

agent.sinks.k1.channel = c1

agent.sinks.k1.type = avro

agent.sinks.k1.hostname = machine-0

agent.sinks.k1.port = 51515

agent.sinks.k2.channel = c1

agent.sinks.k2.type = avro

agent.sinks.k2.hostname = machine-1

agent.sinks.k2.port = 51515

 

配置machine-0，定义文件conf/loadbalance.properties。

 #loadbalance.properties。

agent.sources=s1

agent.channels=c1

agent.sinks=k1

 

agent.sources.s1.type=avro

agent.sources.s1.channels=c1

agent.sources.s1.bind=0.0.0.0

agent.sources.s1.port=51515

 

agent.channels.c1.type=memory

agent.channels.c1.capacity = 10000

agent.channels.c1.transactionCapacity =10000

agent.channels.c1.byteCapacityBufferPercentage= 20

agent.channels.c1.byteCapacity = 800000

 

agent.sinks.k1.type=hdfs

agent.sinks.k1.channel=c1#这台机上没有安装hadoop节点，请配位#hdfs://ip-address:8020/flume/%Y/%m

agent.sinks.k1.hdfs.path=/flume/%Y/%m

agent.sinks.k1.hdfs.filePrefix=flume

agent.sinks.k1.hdfs.fileSuffix=.log

agent.sinks.k1.hdfs.rollInterval=3600

agent.sinks.k1.hdfs.rollCount=0

agent.sinks.k1.hdfs.rollSize=0

agent.sinks.k1.hdfs.fileType=DataStream

agent.sinks.k1.hdfs.writeFormat=Text

agent.sinks.k1.hdfs.useLocalTimeStamp=false

 

与machine-0一样，配置machine-1。

启动flume-ng。进入flume目录下（将flume配置到linux环境变量中，不需要这一步骤，直接启动就可以了），输入下列命令。

bin/flume-ng agent -n agent -c conf –f  \

conf/loadbalance.properties-Dflume.root.logger=DEBUG,console

在三台机上，启动flume。