flume基本用法_the maximum backoff period for the failed sink-CSDN博客

本文链接：https://blog.csdn.net/qq_39906591/article/details/83280833

exec source采集日志agent文件

source type为exec，sink type为logger，channel type为memory

Name the components on this agent

a1.sources = r1
a1.sinks = k1
a1.channels = c1

Describe/configure the source

a1.sources.r1.type = exec
a1.sources.r1.command = tail -f /opt/cdh-5.3.6/flume-1.5.0-cdh5.3.6/conf/flume-test/test.log
a1.sources.tailsource-1.shell = /bin/bash -c

Describe the sink

a1.sinks.k1.type = logger

Use a channel which buffers events in memory

a1.channels.c1.type = memory
a1.channels.c1.capacity = 100
a1.channels.c1.transactionCapacity = 10

Bind the source and sink to the channel

a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1

注：a1为agent name，r1为source name，k1为sink name，c1为channel name

执行命令：bin/flume-ng agent --name a1 --conf conf/ --conf-file conf/flume-test/flume-exec.conf -Dflume.root.logger=INFO,console

flume HA高可用集群agent文件

source type为exec，sink type为logger，channel type为memory，使用failover

搭建的环境：3台虚拟机，主机名分别为：master，slave1，slave2

Master节点的agent文件如下：

Name the components on this agent

a1.sources = r1
a1.sinks = k1 k1
a1.channels = c1

Describe/configure the source

a1.sources.r1.type = exec
a1.sources.r1.command = tail -f /opt/cdh-5.3.6/flume-1.5.0-cdh5.3.6/conf/flume-test/test.log
a1.sources.tailsource-1.shell = /bin/bash -c

Describe the sink

a1.sinks.k1.type = avro
a1.sinks.k1.hostname = slave1
a1.sinks.k1.port = 50020

a1.sinks.k1.type = avro
a1.sinks.k1.hostname = slave2
a1.sinks.k1.port = 50020

failover sink group

a1.sinkgroups = g1
a1.sinkgroups.g1.sinks = k1 k1
a1.sinkgroups.g1.processor.type = failover
a1.sinkgroups.g1.processor.priority.k1 = 10
a1.sinkgroups.g1.processor.priority.k1 = 5
a1.sinkgroups.g1.processor.maxpenalty = 10000 #The maximum backoff period for the failed Sink (in millis)