前言
- 操作系统:CentOS 7
- Java版本:1.8.0_221
- Flume版本:1.8.0
1. Kafka Channel使用场景
- 配合Flume Source、Flume Sink使用,为Event的传输提供一种具有高可用的Channel
- 配合Flume Source和拦截器interceptor使用,无Sink,用于将Flume搜集的Event传输到Kafka集群指定Topic中,便于Kafka消息订阅者使用
- 配合Flume Sink使用,如HDFS Sink、HBaseSink等,无Source,用于提供一种低延迟、高容错的传输Event方式,直接通过Flume Agent将Kafka中的Event传输给Flume Sink
2. Flume Agent配置示例
- 此Flume Agent配置文件主要用于启动Agent监控指定日志文件的更新内容,并将其进行简单的过滤、分类和标记,最终输出到对应的Kafka Topic中。其中使用到的Flume组件有TailDir Source、2个自定义拦截器Interceptor、Channel Selector、2个Kafka Channel等
a1.sources = r1
a1.channels = c1 c2
a1.sources.r1.type = TAILDIR
a1.sources.r1.positionFile = /opt/module/flume-1.8.0/.position/taildir_position.json
a1.sources.r1.filegroups = f1
a1.sources.r1.filegroups.f1 = /tmp/logs/app.+
a1.sources.r1.fileHeader = true
a1.sources.r1.interceptors = i1 i2
a1.sources.r1.interceptors.i1.type = com.tomandersen.flume.interceptor.LogETLInterceptor$Builder
a1.sources.r1.interceptors.i2.type = com.tomandersen.flume.interceptor.LogTypeInterceptor$Builder
a1.sources.r1.selector.type = multiplexing
a1.sources.r1.selector.header = topic
a1.sources.r1.selector.mapping.topic_start = c1
a1.sources.r1.selector.mapping.topic_event = c2
a1.channels.c1.type = org.apache.flume.channel.kafka.KafkaChannel
a1.channels.c1.kafka.bootstrap.servers = kafkaServer1:9092,kafkaServer2:9092,kafkaServer3:9092
a1.channels.c1.kafka.topic = topic_start
a1.channels.c1.parseAsFlumeEvent = false
a1.channels.c2.type = org.apache.flume.channel.kafka.KafkaChannel
a1.channels.c2.kafka.bootstrap.servers = kafkaServer1:9092,kafkaServer2:9092,kafkaServer3:9092
a1.channels.c2.kafka.topic = topic_event
a1.channels.c2.parseAsFlumeEvent = false
a1.sources.r1.channels = c1 c2
End~