负载均衡:
source里的event流经channel,进入sink组,在sink组内部根据负载算法(round_robin、random)选择sink,后续可以选择不同机器上的agent实现负载均衡。
使用JVM 运行Flume。每台机器运行一个agent,但是可以在一个agent中包含多个sources和sinks。
Client,生产数据,运行在一个独立的线程。
集群采用3台机器:
master
slave1
slave2
master 机器配置:
agent.sources=s1
agent.channels=c1agent.sinks=k1 k2
agent.sinkgroups = g1
agent.sinkgroups.g1.sinks = k1 k2
agent.sinkgroups.g1.processor.type =load_balance
agent.sinkgroups.g1.processor.selector =round_robin
agent.sinkgroups.g1.processor.backoff =true
agent.sources.s1.type=avro
agent.sources.s1.channels=c1
agent.sources.s1.bind=0.0.0.0
agent.sources.s1.port=51515
agent.sources.s1.interceptors=i1
agent.sources.s1.interceptors.i1.type=timestamp
agent.channels.c1.type=memory
agent.channels.c1.capacity = 1000
agent.channels.c1.transactionCapacity = 100
agent.sinks.k1.channel = c1agent.sinks.k1.type = avro
agent.sinks.k1.hostname = slave1
agent.sinks.k1.port = 41414
agent.sinks.k2.channel = c1
agent.sinks.k2.type = avro
agent.sinks.k2.hostname = slave2
agent.sinks.k2.port = 41414
source端可以使用exec,直接从host1端就可以采集数据
agent.sources.r1.type = exec
agent.sources.r1.channels=c1
agent.sources.r1.command=tail -F /home/flume/xx.log
slave1机器配置:
agent.channels=c1
agent.sinks=k1
agent.sources.s1.type=avro
agent.sources.s1.channels=c1
agent.sources.s1.bind=0.0.0.0
agent.sources.s1.port=41414
agent.channels.c1.type=memory
agent.channels.c1.capacity = 10000
agent.channels.c1.transactionCapacity =10000
agent.channels.c1.byteCapacityBufferPercentage= 20
agent.channels.c1.byteCapacity = 800000
agent.sinks.k1.type=hdfs
agent.sinks.k1.channel=c1
agent.sinks.k1.hdfs.path=/flume/%Y/%m
agent.sinks.k1.hdfs.filePrefix=flume
agent.sinks.k1.hdfs.fileSuffix=.log
agent.sinks.k1.hdfs.rollInterval=3600
agent.sinks.k1.hdfs.rollCount=0
agent.sinks.k1.hdfs.rollSize=0
agent.sinks.k1.hdfs.fileType=DataStream
agent.sinks.k1.hdfs.writeFormat=Text
agent.sinks.k1.hdfs.useLocalTimeStamp=fals
slave2与slave1 的配置一样。
客户端机器暂时使用flume-ng的agent发送,配置如下:
agent.channels = c1
agent.sources = r1
agent.sinks = k1
agent.sources.r1.channels = c1
agent.sources.r1.type = exec
agent.sources.r1.command = cat /tmp/linux.log
agent.channels.c1.type=memory
agent.channels.c1.capacity = 10000
agent.channels.c1.transactionCapacity =10000
agent.sinks.k1.type = avro
agent.sinks.k1.channel = c1
agent.sinks.k1.hostname = matser
agent.sinks.k1.port = 51515
客户端/tmp/linux.log 文件发送给master。
启动:
启动方式是:先启动slave1 和slave2,然后启动master,最后启动client。
测试过程中,可以随时将slave1 或slave2 停止,过一段时间再启动。
slave1启动:
bin/flume-ng agent -c conf -f conf/load_master.conf -n agent
slave2启动:
bin/flume-ng agent -c conf -f conf/load_slave1.conf -n agent
master启动
bin/flume-ng agent -c conf -f conf/load_slave2.conf -n agent
客户端:
bin/flume-ng agent -c conf -f conf/client.conf -n agent