flume从文本中读取文件到kafka

最新推荐文章于 2023-04-27 17:21:41 发布

lipfff

最新推荐文章于 2023-04-27 17:21:41 发布

阅读量864

点赞数

分类专栏：大数据文章标签： flume kafka

本文链接：https://blog.csdn.net/qq_35008624/article/details/109144121

版权

需求：

监听日志文件的末尾内容变动则把内容发到kafka.

步骤：

1、配置flume的配置文件

flume2kafka.conf

# Name the components on this agent
a1.sources = r1
a1.sinks = k1
a1.channels = c1

# Describe/configure the source
a1.sources.r1.type = exec
a1.sources.r1.command = tail -f /opt/module/logs-flume/bigdata.log

# 设置kafka接收器 
a1.sinks.k1.type = org.apache.flume.sink.kafka.KafkaSink
# 设置kafka的broker地址和端口号
a1.sinks.k1.brokerList=hadoop101:9092,hadoop102:9092,hadoop103:9092
# 设置Kafka的topic
a1.sinks.k1.topic=flume-kafka-1001
# 设置序列化的方式
a1.sinks.k1.serializer.class=kafka.serializer.StringEncoder

# use a channel which buffers events in memory
a1.channels.c1.type=memory
a1.channels.c1.capacity = 100000
a1.channels.c1.transactionCapacity = 1000

# Bind the source and sink to the channel
a1.sources.r1.channels=c1
a1.sinks.k1.channel=c1