windows系统flume数据传给kafka

最新推荐文章于 2024-04-17 08:50:21 发布

hu-Johnson

最新推荐文章于 2024-04-17 08:50:21 发布

阅读量1.2k

点赞数

分类专栏：云计算、大数据

本文链接：https://blog.csdn.net/m0_37847641/article/details/81097144

版权

本文介绍了如何在Windows系统上配置Flume将数据传输到Kafka。首先，详细讲述了安装Zookeeper的步骤，包括设置环境变量和配置zoo.cfg。接着，讲解了启动Kafka服务和创建Topic的操作。最后，指导如何启动Kafka的生产者和消费者进行数据的发送与接收。

摘要由CSDN通过智能技术生成

1、安装zookeeper

更改flume配置为kafka

# Describe the sink
# a1.sinks.k1.type = logger
# 指定Flume sink
a1.sinks.k1.channel = c1
a1.sinks.k1.type = org.apache.flume.sink.kafka.KafkaSink
a1.sinks.k1.kafka.topic = test
a1.sinks.k1.kafka.bootstrap.servers =172.22.20.59:9092
a1.sinks.k1.kafka.flumeBatchSize = 100
a1.sinks.k1.kafka.producer.acks = 1
a1.sinks.k1.kafka.producer.linger.ms = 1
a1.sinks.k1.kafka.producer.compression.type = snappy
a.系统变量中添加ZOOKEEPER_HOME=E:\dashuju\zookeeper-3.5.4-beta

b.编辑系统变量中的path变量，增加%ZOOKEEPER_HOME%\bin

conf 添加文件 zoo.cfg

# The number of milliseconds of each tick 心跳间隔毫秒每次

tickTime=2000

# The number of ticks that the initial

# synchronization phase can take

initLimit=10

# The number of ticks that can pass between

# sending a request and getting anacknowledge