安装:
使用背景
通过logback生成日志 --> flume抓取 --> 发送到kafka。
下载
下载: http://archive.apache.org/dist/flume/
flume版本: apache-flume-1.8.0-bin.tar.gz
解压缩
tar -xvf apache-flume-1.8.0-bin.tar.gz
配置
1、配置java环境变量
cd apache-flume-1.8.0/conf
vi flume-env.sh
(由于我的是mac本,所以用如下写法)
export JAVA_HOME="$(/usr/libexec/java_home)"
2、设置flume 环境变量
vi ~/.bash_profile
FLUME_HOME=/Users/xxx/myapp/apache-flume-1.8.0
export PATH=$FLUME_HOME/bin:xxxx
3、使配置生效
source ~/.bash_profile
4、验证是否安装成功
flume-ng version
xxx@xxxMacBook-Pro bin % flume-ng version
Flume 1.8.0
Source code repository: https://git-wip-us.apache.org/repos/asf/flume.git
Revision: 99f591994468633fc6f8701c5fc53e0214b6da4f
Compiled by denes on Fri Sep 15 14:58:00 CEST 2017
From source with checksum fbb44c8c8fb63a49be0a59e27316833d
使用
Source
Taildir Source
监控指定的多个文件,一旦文件内有新写入的数据,就会将其写入到指定的sink内,本来源可靠性高,不会丢失数据,建议使用;但目前不适用于Windows系统;其不会对于跟踪的文件有任何处理,不会重命名也不会删除,不会做任何修改,这点比Spooling Source有优势;目前不支持读取二进制文件,支持一行一行的读取文本文件;在实时数据处理中,可以用该方式取代Exec方式,因为本方式可靠性高。
配置文件
# source的名字
a1.sources = r1
# channels的名字
a1.channels = c1
# sink的名字
a1.sinks = k1
# 指定source使用的channel
a1.sources.r1.channels = c1
# 指定sink使用的channel
a1.sinks.k1.channel = c1
######## source相关配置 ########
# source类型
a1.sources.r1.type = TAILDIR
# 元数据位置
a1.sources.r1.positionFile = /Users/xxx/logs/flume/taildir_position.json
# 监控的目录
a1.sources.r1.filegroups = f1
a1.sources.r1.filegroups.f1=/Users/xxx/logs/qjFlowFile/.*log
a1.sources.r1.fileHeader = true
######## channel相关配置 ########
# channel类型
a1.channels.c1.type = file
# 数据存放路径
a1.channels.c1.dataDirs = /Users/xxx/logs/flume/filechannle/dataDirs
# 检查点路径
a1.channels.c1.checkpointDir = /Users/xxx/logs/flume/filechannle/checkpointDir
# channel中最多缓存多少
a1.channels.c1.capacity = 1000
# channel一次最多吐给sink多少
a1.channels.c1.transactionCapacity = 100
######## sink相关配置 ########
# sink类型
a1.sinks.k1.type = org.apache.flume.sink.kafka.KafkaSink
# brokers地址
a1.sinks.k1.kafka.bootstrap.servers = cache01:9092,cache02:9092,cache03:9092
# topic
a1.sinks.k1.kafka.topic = flume-demo
# 压缩
a1.sinks.k1.kafka.producer.compression.type = snappy
开启flume
bin/flume-ng agent --conf conf --conf-file conf/flume-kafka.conf --name a1 -Dflume.root.logger=INFO,console
元数据信息
[{"inode":10486453,"pos":10,"file":"/Users/xxx/logs/qjFlowFile/GCSC-D.server1.20210810.0.log"}][{"inode":10486453,"pos":10,"file":"/Users/xxx/logs/qjFlowFile/GCSC-D.server1.20210810.0.log"}][{"inode":10486453,"pos":10,"file":"/Users/xxx/logs/qjFlowFile/GCSC-D.server1.20210810.0.log"}][{"inode":10486453,"pos":10,"file":"/Users/xxx/logs/qjFlowFile/GCSC-D.server1.20210810.0.log"}][{"inode":10486453,"pos":10,"file":"/Users/xxx/logs/qjFlowFile/GCSC-D.server1.20210810.0.log"}][{"inode":10486453,"pos":20,"file":"/Users/xxx/logs/qjFlowFile/GCSC-D.server1.20210810.0.log"}][{"inode":10486453,"pos":20,"file":"/Users/xxx/logs/qjFlowFile/GCSC-D.server1.20210810.0.log"}][{"inode":10486453,"pos":20,"file":"/Users/xxx/logs/qjFlowFile/GCSC-D.server1.20210810.0.log"}]