flume学习

最新推荐文章于 2023-06-09 10:12:02 发布

lei1291

最新推荐文章于 2023-06-09 10:12:02 发布

阅读量224

点赞数

分类专栏： hadoop学习文章标签： flume

本文链接：https://blog.csdn.net/lei1291/article/details/79338634

版权

hadoop学习专栏收录该内容

15 篇文章 0 订阅

订阅专栏

安装

上传安装包apache-flume-1.6.0-bin.tar.gz到数据源所在节点上

然后解压 tar -zxvf apache-flume-1.6.0-bin.tar.gz

然后进入flume的目录，修改conf下的flume-env.sh，在里面配置JAVA_HOME

配置采集方案

监听本地目录输出到logger

vi spool-logger.conf

# Name the components on this agent
a1.sources = r1
a1.sinks = k1
a1.channels = c1

# Describe/configure the source
#监听目录,spoolDir指定目录, fileHeader要不要给文件夹前坠名
a1.sources.r1.type = spooldir
a1.sources.r1.spoolDir = /home/hadoop/flumespool
a1.sources.r1.fileHeader = true

# Describe the sink
a1.sinks.k1.type = logger

# Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100

# Bind the source and sink to the channel
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1

启动

bin/flume-ng agent -c ./conf -f ./conf/spool-logger.conf -n a1 -Dflume.root.logger=INFO,console