flume-kafka整合--实时日志采集

最新推荐文章于 2024-03-09 23:16:05 发布

上善若水211

最新推荐文章于 2024-03-09 23:16:05 发布

阅读量651

点赞数

分类专栏：大数据 flume-kafk 文章标签： flume-kafk

本文链接：https://blog.csdn.net/tuzhihai/article/details/78800560

版权

大数据同时被 2 个专栏收录

8 篇文章 0 订阅

订阅专栏

flume-kafk

1 篇文章 0 订阅

订阅专栏

本文介绍了如何整合Flume和Kafka进行实时日志采集。首先，详细阐述了Flume的exec-source与memory-channel及avro-sink的架构，以及avro-source的使用。接着，讲解了Kafka的启动过程，包括Zookeeper和Kafka服务器的启动。然后，通过模拟数据并启动Kafka消费者监听日志文件，展示了Flume将日志发送到Kafka的流程。最后，提到了配置文件exec-memory-avro.conf和avro-memory-kafka.conf的重要性。

摘要由CSDN通过智能技术生成

flume采用架构

exec-source + memory-channel + avro-sink

avro-source + memory-channel + kafka-sink

kafka采用架构

启动zookeeper

zkServer.sh start

启动kafka(启动一个“篮子”)

kafka-server-start.sh -daemon $KAFKA_HOME/config/server.properties &

模拟

启动一个kafka消费者监听日志文件

kafka-console-consumer.sh –zookeeper 192.168.145.128:2181 –from-beginning –topic firstTopic

向日志文件中添加数据

echo hi,flume-kafka framework >> flume-kafka.txt

exec-memory-avro.conf



flume-ng agent \
  --name exec-memory-avro \
  --conf $FLUME_HOME/conf \
  --conf-file $FLUME_HOME/conf/exec-memory-avro.conf \
  -Dflume.root.logger=INFO,console 

# example exec-memory-avro
exec-memory-avro.sources = exec-source
exec-memory-avro.sinks = avro-sink
exec-memory-avro.channels = memory-channel

# Describe/configure the source
exec-memory-avro.sources.exec-source.type = exec
exec-memory-avro.sources.exec-source.command = tail -F  /root/data/flume-kafka.txt
exec-memory-avro.sources.exec-source.shell = /bin/sh -c

# Describe/ the sink
exec-memory-avro.sinks.avro-sink.type = avro
exec-memory-avro.sinks.avro-sink.hostname = 192.168.145.128
exec-memory-avro.sinks.avro-sink.port = 44444 

# Use a channel which buffers events in memory
exec-memory-avro.channels.memory-channel.type = memory

# Bind the source and sink to the channel
exec-memory-avro.sources.exec-source.channels = memory-channel
exec-memory-avro.sinks.avro-sink.channel = memory-channel

avro-memory-kafka.conf


flume-ng agent \
  --name avro-memory-kafka \
  --conf $FLUME_HOME/conf \
  --conf-file $FLUME_HOME/conf/avro-memory-kafka.conf \
  -Dflume.root.logger=INFO,console 


# example avro-memory-kafka
avro-memory-kafka.sources = avro-source
avro-memory-kafka.sinks = kafka-sink
avro-memory-kafka.channels = memory-channel

# Describe/configure the source
avro-memory-kafka.sources.avro-source.type = avro
avro-memory-kafka.sources.avro-source.bind = 192.168.145.128
avro-memory-kafka.sources.avro-source.port = 44444

# Describe/ the sink
avro-memory-kafka.sinks.kafka-sink.type = org.apache.flume.sink.kafka.KafkaSink
avro-memory-kafka.sinks.kafka-sink.brokerList = 192.168.145.128:9092
avro-memory-kafka.sinks.kafka-sink.topic = firstTopic
avro-memory-kafka.sinks.kafka-sink.batchSize = 3
avro-memory-kafka.sinks.kafka-sink.requiredAcks = 1


# Use a channel which buffers events in memory
avro-memory-kafka.channels.memory-channel.type = memory

# Bind the source and sink to the channel
avro-memory-kafka.sources.avro-source.channels = memory-channel
avro-memory-kafka.sinks.kafka-sink.channel = memory-channel