转载:https://www.jianshu.com/p/9cd850b49dbf
背景 : flume taildir源码只支持监控一级目录下的文件
目标 : 实现目录递归, 即可以同时监控子目录.
定制步骤 :
- 下载flume源码
- 修改flume-taildir-source源码
flume版本: 1.9.2
flume taildir收集监控文件的代码位置:
flume-taildir-source模块 -> TaildirMatcher.java -> getMatchingFilesNoCache方法
新增方法: getMatchingFilesNoCacheRecurseFolder , recurseFolder
List<File> getMatchingFiles() {
long now = TimeUnit.SECONDS.toMillis(
TimeUnit.MILLISECONDS.toSeconds(System.currentTimeMillis()));
long currentParentDirMTime = parentDir.lastModified();
List<File> result;
// calculate matched files if
// - we don't want to use cache (recalculate every time) OR
// - directory was clearly updated after the last check OR
// - last mtime change wasn't already checked for sure
// (system clock hasn't passed that second yet)
if (!cachePatternMatching ||
lastSeenParentDirMTime < currentParentDirMTime ||
!(currentParentDirMTime < lastCheckedTime)) {
// lastMatchedFiles = sortByLastModifiedTime(getMatchingFilesNoCache());
// 使用自己的递归目录方法getMatchingFilesNoCacheRecurseFolder
lastMatchedFiles = sortByLastModifiedTime(getMatchingFilesNoCacheRecurseFolder());
lastSeenParentDirMTime = currentParentDirMTime;
lastCheckedTime = now;
}
private List<File> getMatchingFilesNoCacheRecurseFolder() {
List<File> result = Lists.newArrayList();
List<Path> paths = recurseFolder(parentDir);
for(Path path:paths){
try (DirectoryStream<Path> stream = Files.newDirectoryStream(path, fileFilter)) {
for (Path entry : stream) {
result.add(entry.toFile());
}
} catch (IOException e) {
e.printStackTrace();
}
}
String matchedFileNames = result.stream().map(r-> r.getAbsolutePath()).collect(Collectors.joining("\n"));
logger.debug("============================matched files=======================================");
logger.debug(matchedFileNames);
logger.debug("=============================matched files======================================");
return result;
}
public List<Path> recurseFolder(File root){
List<Path> allParentFolders = new ArrayList<>();
allParentFolders.add(root.toPath());
if (root.exists()) {
File[] files = root.listFiles();
if (null == files || files.length == 0) {
return allParentFolders;
} else {
for (File subFile : files) {
if (subFile.isDirectory()) {
allParentFolders.addAll(recurseFolder(subFile));
}
}
}
}
return allParentFolders;
}
这种方法在设计上来讲, 不够雅观, "/logs/.*.log"这个正则表达式,从任何角度来看都应该只匹配/logs本目录下的文件,不适合表示其子目录. 你可以进一步考虑更完美的实现方式,但是本质实现是类似的.