flume配置-生产环境下 Taildir Source to kafka Sink

最新推荐文章于 2024-08-13 16:26:30 发布

RangeYan2012

最新推荐文章于 2024-08-13 16:26:30 发布

阅读量5.6k

点赞数

分类专栏：原创大数据系统运维笔记环境搭建 Flume

本文链接：https://blog.csdn.net/yanshu2012/article/details/53993206

版权

本文介绍了如何在生产环境中使用Flume从Taildir Source收集后端系统日志，并将这些日志高效地写入Kafka集群，为日志分析提供实时数据流。

摘要由CSDN通过智能技术生成

生产环境下用flume搜集后端系统产产生的日志，并写入kafka集群，可以参照下面配置。

clog.sources = source_log
clog.channels = channel_log
clog.sinks = sink_log1 sink_log2 sink_log3 

clog.sources.source_log.type = TAILDIR
clog.sources.source_log.filegroups =  = f1
#####正则匹配文件路径###### 
clog.sources.source_log.filegroups.f1 = /home/data/log/.*.log 
clog.sources.source_log.skipToEnd = True
clog.sources.source_log.positionFile = /home/data/taildir_position.json	 
clog.sources.source_log.batchSize = 1000
clog.sources.source_log.channels = channel_log

clog.sinks.sink_log1.type = org.apache.flume.sink.kafka.KafkaSink
clog.sinks.sink_log1.kafka.topic = haproxy
clog.sinks.sink_log1.kafka.bootstrap.servers = kafka1:9001,kafka2:9001,kafka3:9001
clog.sinks.sink_log1.flumeBatchSize = 2000
clog.sinks.sink_log1.kafka.producer.acks = 1
clog.sinks.sink_log1.channel = channel_log

clog.sinks.sink_log2.type