1.flume的source选择
生产上一般选择taildir
# 官网参考配置
http://flume.apache.org/releases/content/1.9.0/FlumeUserGuide.html
a1.sources = r1
a1.channels = c1
a1.sources.r1.type = TAILDIR
a1.sources.r1.channels = c1
a1.sources.r1.positionFile = /var/log/flume/taildir_position.json
a1.sources.r1.filegroups = f1 f2
a1.sources.r1.filegroups.f1 = /var/log/test1/example.log
a1.sources.r1.headers.f1.headerKey1 = value1
a1.sources.r1.filegroups.f2 = /var/log/test2/.*log.*
a1.sources.r1.headers.f2.headerKey1 = value2
a1.sources.r1.headers.f2.headerKey2 = value2-2
a1.sources.r1.fileHeader = true
a1.sources.ri.maxBatchCount = 1000
1.1 TAILDIR Souce支持断点还原
positionFile可以记录偏移量,即使flume agent挂掉,重启后可继续上次断点put数据到channel
taildir_position.json 定时刷新,File in JSON format to record the inode, the absolute path and the last position of each tailing file.
[root@hadoop01 flume]# tail -20f taildir_position.json
[{"inode":103011733,"pos":39600,"file":"/var/log/hadoop-hdfs/hadoop-cmf-hdfs-NAMENODE-ruozedata001.log.out"}]tail: taildir_position.json: file truncated
[{"inode":103011733,"pos":39600,"file":"/var/log/hadoop-hdfs/hadoop-cmf-hdfs-NAMENODE-ruozedata001.log.out"}]tail: taildir_position.json: file truncated
[{"inode":103011733,"pos":39600,"file":"/var/log/hadoop-hdfs/hadoop-cmf-hdfs-NAMENODE-ruozedata001.log.out"}]
1.2 可配置文件组,里面使用正则表达式配置多个要监控的文件
a1.sources.r1.filegroups = f1 f2
a1.sources.r1.filegroups.f1 = /var/log/test1/example.log
a1.sources.r1.headers.f1.headerKey1 = value1
a1.sources.r1.filegroups.f2 = /var/log/test2/.*log.*
2. TAILDIR不能覆盖的场景
这么好的taildir source有一点不完美,不能支持递归监控文件夹。
比如conf文件中配置了:
# Describe/configure the dirsource
a1.sources.r1.type = TAILDIR
a1.sources.r1.positionFile = /var/log/flume/taildir_position.json
a1.sources.r1.filegroups = f1
a1.sources.r1.filegroups.f1 = /var/log/hadoop-hdfs/.*log.*
此时可以监控到/var/log/hadoop-hdfs/test.log 的日志文件
但是文件路径/var/log/hadoop-hdfs/f1/test.log
/var/log/hadoop-hdfs/f1/f2/test.log
......
是监控不到的
3. 修改源代码,使得flume支持递归监控文件夹。
所以就只能修改源代码了,需要注意的是无论是Apache版本的还是CDH的都能够兼容使用,我这里使用的版本是apache-flume-1.7.0-bin,但是即使你使用cdh的版本编译源码,也是没问题的。
3.1 flume taildir source源码结构
在Module flume-taildir-source下
org.apache.flume.source.taidir
-- ReliableTaildirEventReader 监听文件及获取文件的position
-- TaildirMatcher Identifies and caches the files matched by single file pattern for {@code TAILDIR} source.
-- TaildirSource 核心类,configure-> start-> process-> stop 四大重要方法
-- TaildirSourceConfigurationConstants TaildirSource的常量配置类
-- TailFile TailFile对象(构造及get set方法),其他类中使用该类
3.2 TaildirSource 核心类解析
TaildirSource 核心类 顺序 configure-> start-> process-> stop
public class TaildirSource extends AbstractSource implements
PollableSource, Configurable {
//2.调用start,构建出ReliableTaildirEventReader 监听文件及获取文件的position
@Override
public synchronized void start() {
}
//4.stop
@Override
public synchronized void stop() {
}
//1.首先加载配置文件
@Override
public synchronized void configure(Context context) {
}
//3.【改源码的重点】中间处理过程
@Override
public Status process() {
}
}
TaildirSource继承了PollableSource,从PollableSource的源码注释可以看到
是一个外部驱动定时拉取file的数据。这就是sparkStreaming一样,是伪实时的,而不是像flink,是真正的实时。
/**
* A {@link Source} that requires an external driver to poll to determine
* whether there are {@linkplain Event events} that are available to ingest
* from the source.
*
* @see org.apache.flume.source.EventDrivenSourceRunner
*/
public interface PollableSource extends Source {
}
3.2.1 start方法会使用建造者模式 构建创建一个ReliableTaildirEventReade
# 建造者模式详解
https://www.runoob.com/design-pattern/builder-pattern.html
@Override
public synchronized void start() {
logger.info("{} TaildirSource source starting with directory: {}", getName(), filePaths);
try {
reader = new ReliableTaildirEventReader.Builder()
.filePaths(filePaths)
.headerTable(headerTable)
.positionFilePath(positionFilePath)
.skipToEnd(skipToEnd)
.addByteOffset(byteOffsetHeader)
.cachePatternMatching(cachePatternMatching)
.recursive(isRecursive)
.annotateFileName(fileHeader)
.fileNameHeader(fileHeaderKey)
.build();
} catch (IOException e) {
throw new FlumeException("Error instantiating ReliableTaildirEventReader", e);
}
idleFileChecker = Executors.newSingleThreadScheduledExecutor(
new ThreadFactoryBuilder().setNameFormat("idleFileChecker").build());
idleFileChecker.scheduleWithFixedDelay(new idleFileCheckerRunnable(),
idleTimeout, checkIdleInterval, TimeUnit.MILLISECONDS);
positionWriter = Executors.newSingleThreadScheduledExecutor(
new ThreadFactoryBuilder().setNameFormat("positionWriter").build());
positionWriter.scheduleWithFixedDelay(new PositionWriterRunnable(),
writePosInitDelay, writePosInterval, TimeUnit.MILLISECONDS);
super.start();
logger.debug("TaildirSource started");
sourceCounter.start();
}
3.2.2 【改源码重点】process方法
# 按照下面路径找到需要修改的方法
org.apache.flume.source.taildir.TaildirSource的process方法
-- reader.updateTailFiles()
-- updateTailFiles(false) 获取监控文件
-- taildir.getMatchingFiles() 根据配置获取匹配文件
-- sortByLastModifiedTime(getMatchingFilesNoCache()) 根据文件最新更新时间确定是否需要tail
-- getMatchingFilesNoCache() 需要修改的方法
@Override
public Status process() {
Status status = Status.READY;
try {
existingInodes.clear();
//updateTailFiles() 点进去
existingInodes.addAll(reader.updateTailFiles());
for (long inode : existingInodes) {
TailFile tf = reader.getTailFiles().get(inode);
if (tf.needTail()) {
tailFileProcess(tf, true);
}
}
closeTailFiles();
try {
TimeUnit.MILLISECONDS.sleep(retryInterval);
} catch (InterruptedException e) {
logger.info("Interrupted while sleeping");
}
} catch (Throwable t) {
logger.error("Unable to tail files", t);
status = Status.BACKOFF;
}
return status;
}
==》
从reader.updateTailFiles()获取需要监控的文件,然后对每一个进行处理,查看最后修改时间,判定是否需要tail,需要tail就tail
那么进入reader.updateTailFiles()
public List<Long> updateTailFiles() throws IOException {
return updateTailFiles(false);
}
==》
/**
* Update tailFiles mapping if a new file is created or appends are detected
* to the existing file.
*/
public List<Long> updateTailFiles(boolean skipToEnd) throws IOException {
updateTime = System.currentTimeMillis();
List<Long> updatedInodes = Lists.newArrayList();
for (TaildirMatcher taildir : taildirCache) {
Map<String, String> headers = headerTable.row(taildir.getFileGroup());
//taildir.getMatchingFiles() 获取匹配的文件 点进去
for (File f : taildir.getMatchingFiles()) {
long inode = getInode(f);
TailFile tf = tailFiles.get(inode);
if (tf == null || !tf.getPath().equals(f.getAbsolutePath())) {
long startPos = skipToEnd ? f.length() : 0;
tf = openFile(f, headers, inode, startPos);
} else {
boolean updated = tf.getLastUpdated() < f.lastModified();
if (updated) {
if (tf.getRaf() == null) {
tf = openFile(f, headers, inode, tf.getPos());
}
if (f.length() < tf.getPos()) {
logger.info("Pos " + tf.getPos() + " is larger than file size! "
+ "Restarting from pos 0, file: " + tf.getPath() + ", inode: " + inode);
tf.updatePos(tf.getPath(), inode, 0);
}
}
tf.setNeedTail(updated);
}
tailFiles.put(inode, tf);
updatedInodes.add(inode);
}
}
return updatedInodes;
}
==》
遍历每一个正则表达式匹配对应的匹配器,每个匹配器去获取匹配的文件!
getMatchingFiles()
List<File> getMatchingFiles() {
long now = TimeUnit.SECONDS.toMillis(
TimeUnit.MILLISECONDS.toSeconds(System.currentTimeMillis()));
long currentParentDirMTime = parentDir.lastModified();
List<File> result;
// calculate matched files if
// - we don't want to use cache (recalculate every time) OR
// - directory was clearly updated after the last check OR
// - last mtime change wasn't already checked for sure
// (system clock hasn't passed that second yet)
if (!cachePatternMatching ||
lastSeenParentDirMTime < currentParentDirMTime ||
!(currentParentDirMTime < lastCheckedTime)) {
//根据文件最新更新时间确定是否需要tail
lastMatchedFiles = sortByLastModifiedTime(getMatchingFilesNoCache());
lastSeenParentDirMTime = currentParentDirMTime;
lastCheckedTime = now;
}
return lastMatchedFiles;
}
== > 找到需要改造的方法
private List<File> getMatchingFilesNoCache() {
List<File> result = Lists.newArrayList();
try (DirectoryStream<Path> stream = Files.newDirectoryStream(parentDir.toPath(), fileFilter)) {
for (Path entry : stream) {
result.add(entry.toFile());
}
} catch (IOException e) {
logger.error("I/O exception occurred while listing parent directory. " +
"Files already matched will be returned. " + parentDir.toPath(), e);
}
return result;
}
3.3 修改源码
3.3.1 重载getMatchingFilesNoCache方法
package org.apache.flume.source.taildir下增加getMatchingFilesNoCache(boolean recursion)方法
**
* 重载getMatchingFilesNoCache方法
* @param recursion whether to open recursion
* @return List of files matching the pattern unsorted
*/
private List<File> getMatchingFilesNoCache(boolean recursion) {
System.out.println("====execution override method================");
if (!recursion) {
//如果传入recursion为false,则调用原方法
return getMatchingFilesNoCache();
}
List<File> result = Lists.newArrayList();
//使用非递归方式遍历文件(将树转成队列),此处使用递归方式遍历也可,但是效率可能不高,具体可见参考资料
Queue<File> dirs = new ArrayBlockingQueue<>(10);
//放parentDir
dirs.offer(parentDir);
//遍历ArrayBlockingQueue中的file
while (dirs.size() > 0) {
//拿parentDir
File dir = dirs.poll();
try {
DirectoryStream<Path> stream = Files.newDirectoryStream(dir.toPath(), fileFilter);
for (Path entry : stream) {
//将文件加入到List
result.add(entry.toFile());
}
} catch (IOException e) {
logger.error("I/O exception occurred while listing parent directory. " +
"Files already matched will be returned. (recursion)" + parentDir.toPath(), e);
}
File[] dirList = dir.listFiles();
if (dirList != null) {
for (File file : dirList) {
if (file.isDirectory()) {
//如果是目录则继续迭代
dirs.add(file);
}
}
}
}
return result;
}
3.3.2 使recursive通过配置载入
(1)TaildirSourceConfigurationConstants 配置类中引入RECURSIVE和DEFAULT_RECURSIVE常量
package org.apache.flume.source.taildir;
public class TaildirSourceConfigurationConstants {
/*** Whether to support recursion. */
public static final String RECURSIVE = "recursive";
public static final boolean DEFAULT_RECURSIVE = false;
......
}
(2)TaildirSource中增加成员变量isRecursive并初始化
public class TaildirSource extends AbstractSource implements
PollableSource, Configurable {
//增加成员变量
public static boolean isRecursive;
......
@Override
public synchronized void configure(Context context) {
//初始化
isRecursive =context.getBoolean(RECURSIVE, DEFAULT_RECURSIVE)
......
}
......
}
(3)使用重载方法getMatchingFilesNoCache(TaildirSource.isRecursive)并载入配置
package org.apache.flume.source.taildir;
public class TaildirMatcher {
List<File> getMatchingFiles() {
long now = TimeUnit.SECONDS.toMillis(
TimeUnit.MILLISECONDS.toSeconds(System.currentTimeMillis()));
long currentParentDirMTime = parentDir.lastModified();
List<File> result;
// calculate matched files if
// - we don't want to use cache (recalculate every time) OR
// - directory was clearly updated after the last check OR
// - last mtime change wasn't already checked for sure
// (system clock hasn't passed that second yet)
if (!cachePatternMatching ||
lastSeenParentDirMTime < currentParentDirMTime ||
!(currentParentDirMTime < lastCheckedTime)) {
/**
* 使用重载方法getMatchingFilesNoCache(TaildirSource.isRecursive)
*/
lastMatchedFiles = sortByLastModifiedTime(getMatchingFilesNoCache(TaildirSource.isRecursive));
lastSeenParentDirMTime = currentParentDirMTime;
lastCheckedTime = now;
}
return lastMatchedFiles;
}
}
4.打包代码并测试
4.1 打包
clean package -DskipTests
flume-1.7.0\flume-ng-sources\flume-taildir-source\target下找到编译好的包
替换原来的flume lib下的flume-taildir-source***.jar
4.2 flume 的agent config配置
vim dirsource_memory_kafka.properties
# Name the components on this agent
a1.sources = r1
a1.sinks = k1
a1.channels = c1
# Describe/configure the dirsource
a1.sources.r1.type = TAILDIR
a1.sources.r1.positionFile = /var/log/flume/taildir_position.json
a1.sources.r1.filegroups = f1
a1.sources.r1.filegroups.f1 = /var/log/hadoop-hdfs/.*log.*
# 开启 递归监控文件夹
a1.sources.r1.recursive = true
# Describe the sink
a1.sinks.k1.type = org.apache.flume.sink.kafka.KafkaSink
a1.sinks.k1.kafka.topic = PREWARNING
a1.sinks.k1.kafka.bootstrap.servers = hadoop01:9092,hadoop02:9092,hadoop03:9092
a1.sinks.k1.kafka.flumeBatchSize = 6000
a1.sinks.k1.kafka.producer.acks = all
a1.sinks.k1.kafka.producer.linger.ms = 1
a1.sinks.ki.kafka.producer.compression.type = snappy
# Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.keep-alive = 90
a1.channels.c1.capacity = 2000000
a1.channels.c1.transactionCapacity = 6000
# Bind the source and sink to the channel
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1
4.3 封装启动脚本
vim start_dirsource_memory_kafka.sh
#!/bin/bash
nohup /flume/flume/bin/flume-ng agent \
-c /flume/flume/conf \
-f /flume/flumeFlu/conf/dirsource_memory_kafka.properties \
-n a1 -Dflume.root.logger=INFO,console &
4.4 启动脚本并运行
(1)查看启动日志,执行了重载方法的输出,说明运行了
(2)模拟生成日志
[root@hadoop01 log]# echo "666" >> ./hadoop-hdfs/hadoop-cmf-hdfs-NAMENODE-hadoop01.log.out
[root@hadoop01 log]# echo "777" >> hadoop-hdfs/test/test.log
(3)在kafak的消费者可以看到结果
afka-console-consumer.sh --topic 'PREWARNING' --bootstrap-server hadoop01:9092,hadoop02:9092,hadoop03:9092 --skip-message-on-error
666
777
参考资料
生产上,修改Flume源码使taildir source支持递归(可配置)
https://mp.weixin.qq.com/s?__biz=MzA5ODY0NzgxNA==&mid=2247486109&idx=1&sn=67de70c93d2d7659dc7182ff20116325&chksm=908f20f4a7f8a9e2bf4ee8d54fb67cec33f53ed064044c0f7043d461cc9495da56ee3c96be06&mpshare=1&scene=1&srcid=&sharer_sharetime=1590537356240&sharer_shareid=de261d3248f7a54980859a381c24c116&key=b045cd156f7027c5ddd185d0bcc106fb6b7e18c5b379f75d9b75b9a1ff125f4eb6a4091f160cd9901b85827b2e8406ce20926ce3161441add48e4d700dca806082fd8f7bbfb0bfc612e4eb54e273b673&ascene=1&uin=MTQxODcwNzI2MA%3D%3D&devicetype=Windows+10&version=62080079&lang=zh_CN&exportkey=Abzk3oDMK0AFVaNEMYgbFGw%3D&pass_ticket=dpT1nm%2FwrkXBvC36UjY6Dz2FG3R1uOC%2Bi76nouZiPqP1ZoKA9BME%2B15kKqrmIDCu
Flume TaildirSource 实现递归
https://blog.csdn.net/qq_38976805/article/details/93117865