Flume安装使用

最新推荐文章于 2024-07-23 08:03:31 发布

suiyin_nj

最新推荐文章于 2024-07-23 08:03:31 发布

阅读量148

点赞数

分类专栏：大数据

本文链接：https://blog.csdn.net/suiyin_nj/article/details/79497013

版权

大数据专栏收录该内容

5 篇文章 0 订阅

订阅专栏

第一步配置环境变量

vim /etc/profile

export FLUME_HOME=/usr/local/flume

export PATH=$PATH:$FLUME_HOME/bin:

source /etc/profile

配置后检查Flume 版本号

flume-ng version

第二步配置flume数据采集配置(日志采集到kafka)playtime.conf

a1.sources=r2
a1.channels=c2

a1.sinks=k2

    #a1.sources.r2.type=exec
    #a1.sources.r2.command=tail -F /data/logs/play/playTime.log
    #a1.sources.r2.shell = /bin/bash -c

#a1.sources.r2.channels=c2

    a1.sources.r2.type = TAILDIR
    a1.sources.r2.positionFile = /data/flume/play/taildir_position.json
    a1.sources.r2.filegroups = f1
    a1.sources.r2.filegroups.f1 = /data/logs/play/playtime.log
    a1.sources.r2.fileHeader = true
    a1.sources.r2.channels = c2

    a1.channels.c2.type=memory
    a1.channels.c2.capacity=1000
    a1.channels.c2.transactioncapacity=100

    a1.sinks.k2.channel = c2
    a1.sinks.k2.type = org.apache.flume.sink.kafka.KafkaSink
    a1.sinks.k2.kafka.topic = tp_playtime
    a1.sinks.k2.kafka.bootstrap.servers = 192.168.1.115:9092

第三步运行运行Flume

nohup flume-ng agent -c conf/ -f conf/playtime.conf -n a1 -Dflume.root.logger=INFO,console >>logs/playtime.log 2>&1 &