Flume 三：监测本地目录写入日志

本文用到的文件会上传以供下载练习
https://download.csdn.net/download/cai_and_luo/12441018
https://download.csdn.net/download/cai_and_luo/12440889

一：创建目录

在/opt下面创建 Flume 要监测的目录 /opt/flumelog/events

[root@cai flumelog]# mkdir events

创建监查点的目录 /opt/flumelog/checkpoint/events

[root@cai flumelog]# mkdir checkpoint

[root@cai checkpoint]# mkdir events

创建结果输出目录 /opt/flumelog/data/events

[root@cai flumelog]# mkdir data
[root@cai flumelog]# cd data/
[root@cai data]# mkdir events
[root@cai data]# pwd
/opt/flumelog/data

二：创建配置文件

在目录 /opt/bigdata/flume/conf/job 下创建各种配置文件

创建配置文件 /events-flume-logger.conf

[root@cai job]# touch ./events-flume-logger.conf
[root@cai job]# ls
events-flume-logger.conf  netcat-flume-logger.conf

进行配置

vi ./events-flume-logger.conf

events.sources = eventsSource
events.channels = eventsChannel
events.sinks = eventsSink

events.sources.eventsSource.type = spooldir
events.sources.eventsSource.spoolDir = /opt/flumelog/events
events.sources.eventsSource.includePattern = events_[0-9]{4}-[0-9]{2}-[0-9]{2}.csv
events.sources.eventsSource.deserializer = LINE
events.sources.eventsSource.deserializer.maxLineLength = 32000

events.channels.eventsChannel.type = file
events.channels.eventsChannel.checkpointDir = /opt/flumelog/checkpoint/events
events.channels.eventsChannel.dataDirs = /opt/flumelog/data/events

events.sinks.eventsSink.type = logger

events.sinks.eventsSink.channel = eventsChannel
events.sources.eventsSource.channels = eventsChannel 

三：启动

./bin/flume-ng agent -c conf/ -f conf/job/events-flume-logger.conf -n events -Dflume.root.logger=INFO,console

离开此shell 页面，去往新 shell 拷贝要监测的文件到待监测目录

四：把要用的文件放入 Linux 本地（用xftp进行拖拽）

在此 /opt/bigdata/flume/conf/events 路径中创建目录，将要用的文件拉进来

[root@cai conf]# mkdir events
[root@cai conf]# cd events/
[root@cai events]# pwd
/opt/bigdata/flume/conf/events

将要进行监测的文件 拷贝 到待监测目录

[root@cai events]# pwd
/opt/bigdata/flume/conf/events
[root@cai events]# cp ./events.csv /opt/flumelog/events/events_2020-05-20.csv

注意：以上 events_2020-05-20.csv 的文件格式是固定的，由以上 /events-flume-logger.conf 的配置文件决定（见下图）

拷贝以后，去往刚刚运行启动代码的 shell 页面，看到正在读数据即成功