SS00005.EBpositions——|Hadoop&EbProjec电商实时数仓项目.v05|——|EbProject.v05|Flume配置&Flume配置|

本文档展示了如何配置Apache Flume event-conf.properties文件，将数据从tail -F命令读取的日志文件中传输到Kafka topic。配置包括设置Flume源、接收器和通道，并详细说明了启动Flume Agent的命令。通过这种方式，可以实现实时监控日志文件并将数据流式传输到Kafka集群。

摘要由CSDN通过智能技术生成

一、Flume配置文件：

### --- flume配置文件：event-conf.properties

~~~     # 写入Flume配置文件：event-conf.properties
[root@hadoop02 ~]# vim /opt/yanqi/servers/flume-1.9.0/conf/event-conf.properties

# the core components
a1.sources = r1
a1.sinks = k1
a1.channels = c1

# set source
a1.sources.r1.type = exec
a1.sources.r1.command = tail -F /opt/yanqi/servers/flume-1.9.0/flumelog/eventlog.log

# Describe the sink
a1.sinks.k1.type = org.apache.flume.sink.kafka.KafkaSink
a1.sinks.k1.kafka.topic=eventlog
a1.sinks.k1.kafka.bootstrap.servers=hadoop01:9092,hadoop02:9092,hadoop03:9092

# Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 20000
a1.channels.c1.transactionCapacity = 10000

# Bind the source and sink to the channel
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1

### --- 数据通过flume把数据传输到控制台

~~~     # 启动Flume Agent文件可以用重命名的方式触发监听
[root@hadoop02 ~]# flume-ng agent \
-c /opt/yanqi/servers/flume-1.9.0/ \
-f /opt/yanqi/servers/flume-1.9.0/conf/event-conf.properties \
-n a1 -Dflume.root.logger=INFO,console 
~~~     # 监听数据

### --- 数据通过flume把数据传输到kafka的eventlog中

~~~     # 启动Flume Agent，从kafka中采集数据并写入到eventlog.log文件中：文件可以用重命名的方式触发监听
[root@hadoop02 ~]# flume-ng agent \
-c /opt/yanqi/servers/flume-1.9.0/ \
-f /opt/yanqi/servers/flume-1.9.0/conf/event-conf.properties \
-n a1 -Dflume.root.logger=INFO,console \
a1.sources.r1.command = tail -F /opt/yanqi/servers/flume-1.9.0/flumelog/eventlog.log