Flume Agent之一个source两个channel两个sink配置

最新推荐文章于 2022-08-22 22:31:31 发布

ENUEI

最新推荐文章于 2022-08-22 22:31:31 发布

阅读量1.8k

点赞数 1

分类专栏：学习记录文章标签： flume agnet

本文链接：https://blog.csdn.net/enuei/article/details/85318309

版权

本文介绍了如何配置Flume Agent，实现一个Source同时将数据发送到两个不同的Channel，再通过两个Sink分别将数据同步到Kafka和HDFS。配置中详细设置了Source的目录监控，两个Memory Channel的容量，以及Sink的Kafka连接参数和HDFS写入参数。

摘要由CSDN通过智能技术生成

一个source 两个channel 两个sink，每个sink 输出内容一致

flume_kafka_hdfs.sources = r1
flume_kafka_hdfs.channels = c1 c2
flume_kafka_hdfs.sinks = k1 k2

#配置sources,监控一个目录
flume_kafka_hdfs.sources.r1.type = spooldir
flume_kafka_hdfs.sources.r1.spoolDir = /home/sivan/flume/spooldir
flume_kafka_hdfs.sources.r1.selector.type=replicating
flume_kafka_hdfs.sources.r1.channels=c1 c2

#配置channel1
flume_kafka_hdfs.channels.c1.type = memory
flume_kafka_hdfs.channels.c1.capacity = 10000

#配置channel2
flume_kafka_hdfs.channels.c2.type = memory
flume_kafka_hdfs.channels.c2.capacity = 10000

#flume_kafka_hdfs.channels.c2.type = file
#flume_kafka_hdfs.channels.c2.checkpointDir = /home/sivan/flume/checkpoint
#flume_kafka_hdfs.channels.c2.dataDirs = /home/sivan/flume/data

#配置sink1,数据从channel1获取,传入kafka
flume_kafka_