flume安装及使用

安装：

使用背景

通过logback生成日志 --> flume抓取 --> 发送到kafka。

下载

下载： http://archive.apache.org/dist/flume/
flume版本： apache-flume-1.8.0-bin.tar.gz

解压缩

tar -xvf apache-flume-1.8.0-bin.tar.gz

配置

1、配置java环境变量

cd apache-flume-1.8.0/conf
vi flume-env.sh 

（由于我的是mac本，所以用如下写法）
export JAVA_HOME="$(/usr/libexec/java_home)"

2、设置flume 环境变量
vi ~/.bash_profile
FLUME_HOME=/Users/xxx/myapp/apache-flume-1.8.0
export PATH=$FLUME_HOME/bin:xxxx
3、使配置生效
source ~/.bash_profile
4、验证是否安装成功
flume-ng version

xxx@xxxMacBook-Pro bin % flume-ng version
Flume 1.8.0
Source code repository: https://git-wip-us.apache.org/repos/asf/flume.git
Revision: 99f591994468633fc6f8701c5fc53e0214b6da4f
Compiled by denes on Fri Sep 15 14:58:00 CEST 2017
From source with checksum fbb44c8c8fb63a49be0a59e27316833d

使用

Source

Taildir Source

监控指定的多个文件，一旦文件内有新写入的数据，就会将其写入到指定的sink内，本来源可靠性高，不会丢失数据，建议使用；但目前不适用于Windows系统；其不会对于跟踪的文件有任何处理，不会重命名也不会删除，不会做任何修改，这点比Spooling Source有优势；目前不支持读取二进制文件，支持一行一行的读取文本文件；在实时数据处理中，可以用该方式取代Exec方式，因为本方式可靠性高。

配置文件

# source的名字
a1.sources = r1
# channels的名字
a1.channels = c1
# sink的名字
a1.sinks = k1

# 指定source使用的channel
a1.sources.r1.channels = c1
# 指定sink使用的channel
a1.sinks.k1.channel = c1

######## source相关配置 ########
# source类型
a1.sources.r1.type = TAILDIR
# 元数据位置
a1.sources.r1.positionFile = /Users/xxx/logs/flume/taildir_position.json
# 监控的目录
a1.sources.r1.filegroups = f1
a1.sources.r1.filegroups.f1=/Users/xxx/logs/qjFlowFile/.*log
a1.sources.r1.fileHeader = true

######## channel相关配置 ########
# channel类型
a1.channels.c1.type = file
# 数据存放路径
a1.channels.c1.dataDirs = /Users/xxx/logs/flume/filechannle/dataDirs
# 检查点路径
a1.channels.c1.checkpointDir = /Users/xxx/logs/flume/filechannle/checkpointDir
# channel中最多缓存多少
a1.channels.c1.capacity = 1000
# channel一次最多吐给sink多少
a1.channels.c1.transactionCapacity = 100


######## sink相关配置 ########
# sink类型
a1.sinks.k1.type = org.apache.flume.sink.kafka.KafkaSink
# brokers地址
a1.sinks.k1.kafka.bootstrap.servers = cache01:9092,cache02:9092,cache03:9092
# topic
a1.sinks.k1.kafka.topic = flume-demo
# 压缩
a1.sinks.k1.kafka.producer.compression.type = snappy

开启flume

bin/flume-ng agent --conf conf --conf-file conf/flume-kafka.conf --name a1 -Dflume.root.logger=INFO,console

元数据信息

[{"inode":10486453,"pos":10,"file":"/Users/xxx/logs/qjFlowFile/GCSC-D.server1.20210810.0.log"}][{"inode":10486453,"pos":10,"file":"/Users/xxx/logs/qjFlowFile/GCSC-D.server1.20210810.0.log"}][{"inode":10486453,"pos":10,"file":"/Users/xxx/logs/qjFlowFile/GCSC-D.server1.20210810.0.log"}][{"inode":10486453,"pos":10,"file":"/Users/xxx/logs/qjFlowFile/GCSC-D.server1.20210810.0.log"}][{"inode":10486453,"pos":10,"file":"/Users/xxx/logs/qjFlowFile/GCSC-D.server1.20210810.0.log"}][{"inode":10486453,"pos":20,"file":"/Users/xxx/logs/qjFlowFile/GCSC-D.server1.20210810.0.log"}][{"inode":10486453,"pos":20,"file":"/Users/xxx/logs/qjFlowFile/GCSC-D.server1.20210810.0.log"}][{"inode":10486453,"pos":20,"file":"/Users/xxx/logs/qjFlowFile/GCSC-D.server1.20210810.0.log"}]