flume failover

Architecture like that:

 

Master conf:

# Name the components on this agent
a1.sources = r1
a1.sinks = k1 k2
a1.channels = c1

#describe/configure the source
a1.sources.r1.type = exec
a1.sources.r1.command = tail -f /mnt/hgfs/share/flume_test/log/logserver.                                                                                    log

#define a memory channel called c1 on a1
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100

#Describe the sink
a1.sinks.k1.type = avro
a1.sinks.k1.hostname = slave1
a1.sinks.k1.port = 52020

a1.sinks.k2.type = avro
a1.sinks.k2.hostname = slave2
a1.sinks.k2.port = 52020

#Bind the source and sink to the channel
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1
a1.sinks.k2.channel = c1

#define a sinkgroup which include k1,k2
a1.sinkgroups = g1
a1.sinkgroups.g1.sinks = k1 k2
a1.sinkgroups.g1.processor.type = failover
a1.sinkgroups.g1.processor.priority.k1 = 1
a1.sinkgroups.g1.processor.priority.k2 = 2
a1.sinkgroups.g1.processor.priority.maxpenality = 10000

Slave1 conf:

# Name the components on this agent
a1.sources = r1
a1.sinks = k1
a1.channels = c1

#define a memory channel called c1 on a1
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100

#Describe the sink
a1.sources.r1.type = avro
a1.sources.r1.bind = slave1
a1.sources.r1.port = 52020

#Bind the source and sink to the channel
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1

a1.sinks.k1.type = logger

Slave2 conf:

# Name the components on this agent
a1.sources = r1
a1.sinks = k2
a1.channels = c1

#define a memory channel called c1 on a1
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100
 

#Describe the sink
a1.sources.r1.type = avro
a1.sources.r1.bind = slave2
a1.sources.r1.port = 52020

#Bind the source and sink to the channel
a1.sources.r1.channels = c1
a1.sinks.k2.channel = c1

a1.sinks.k2.type = logger

 

OK, everything is ready, you can start flume and have a test.

 

Notice:

1. conf file can be modified dynamic;

2.the higher value of a1.sinkgroups.g1.processor.priority means which will be used to collect info, when the higher one change to dead state then the lower one will be used.