flume load_balance

This article describes how to config flume load balance, Master is the data generator, flume agent get logserver.log then transmit to slave1 and slave2, which is the collector, and save log to hdfs.

Master conf:

# Name the components on this agent
a1.sources = r1
a1.sinks = k1 k2
a1.channels = c1

#describe/configure the source
a1.sources.r1.type = exec
a1.sources.r1.command = tail -f /mnt/hgfs/share/flume_test/log/logserver.log

#define a memory channel called c1 on a1
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100

#Describe the sink
a1.sinks.k1.type = avro
a1.sinks.k1.hostname = slave1
a1.sinks.k1.port = 52020

a1.sinks.k2.type = avro
a1.sinks.k2.hostname = slave2
a1.sinks.k2.port = 52020

#Bind the source and sink to the channel
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1
a1.sinks.k2.channel = c1

#define a sinkgroup which include k1,k2
a1.sinkgroups = g1
a1.sinkgroups.g1.sinks = k1 k2
a1.sinkgroups.g1.processor.type = load_balance
a1.sinkgroups.g1.processor.selector = random   --you can choose random or round-robin

Slave1 conf:

# Name the components on this agent
a1.sources = r1
a1.sinks = k1
a1.channels = c1

#describe/configure the source

a1.sources.r1.type = avro
a1.sources.r1.bind = slave1
a1.sources.r1.port = 52020

a1.sources.r1.channels = c1

#define a memory channel called c1 on a1
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100

#Describe the sink

a1.sinks.k1.type = hdfs
a1.sinks.k1.channel = c1
a1.sinks.k1.hdfs.path = hdfs://master:9000/flume_data_pool
a1.sinks.k1.hdfs.filePrefix = events-
a1.sinks.k1.hdfs.filetype = DataStream
a1.sinks.k1.hdfs.writeFormat = Text
a1.sinks.k1.hdfs.rollSize = 0
a1.sinks.k1.hdfs.rollCount = 60000
a1.sinks.k1.hdfs.rollInterval = 600
 

Slave2 conf:

# Name the components on this agent
a1.sources = r1
a1.sinks = k2
a1.channels = c1

#describe/configure the source
a1.sources.r1.type = avro
a1.sources.r1.bind = slave2
a1.sources.r1.port = 52020

a1.sources.r1.channels = c1

#define a memory channel called c1 on a1
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100

#Describe the sink

a1.sinks.k2.type = hdfs
a1.sinks.k2.channel = c1
a1.sinks.k2.hdfs.path = hdfs://master:9000/flume_data_pool
a1.sinks.k2.hdfs.filePrefix = events-
a1.sinks.k2.hdfs.filetype = DataStream
a1.sinks.k2.hdfs.writeFormat = Text
a1.sinks.k2.hdfs.rollSize = 0
a1.sinks.k2.hdfs.rollCount = 60000
a1.sinks.k2.hdfs.rollInterval = 600