| centos | 7 | linux系统发行版 |
| jdk | 1.8 | java开发工具集 |
| hadoop | 2.10.0 | 大数据生态基础组件 |
| zookeeper | 3.5.7 | 分布式应用程序协调服务 |
| kafka | 3.0 | 分布式mq组件 |
| flume | 1.9.0 | 分布式采集传输组件 |
2 报错
- 场景1:动态监控目录多个日志变化,通过flume采集传输到kafka
- 报错日志
org.apache.flume.FlumeException: Error creating positionFile parent directories
at org.apache.flume.source.taildir.TaildirSource.configure(TaildirSource.java:170)
at org.apache.flume.conf.Configurables.configure(Configurables.java:41)
at org.apache.flume.node.AbstractConfigurationProvider.loadSources(AbstractConfigurationProvider.java:325)
at org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:105)
at org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:145)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
Caused by: java.nio.file.FileAlreadyExistsException: /export/server/flume
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:88)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:384)
at java.nio.file.Files.createDirectory(Files.java:674)
at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781)
at java.nio.file.Files.createDirectories(Files.java:727)
at org.apache.flume.source.taildir.TaildirSource.configure(TaildirSource.java:168)
... 11 more
- conf文件如下
#定义组件
a1.sources = r1
a1.channels = c1
#配置source
a1.sources.r1.type = TAILDIR
a1.sources.r1.filegroups = f1
a1.sources.r1.filegroups.f1 = /opt/module/applog/log/app.*
a1.sources.r1.positionFile = /export/server/flume/taildir_position.json
#配置channel
a1.channels.c1.type = org.apache.flume.channel.kafka.KafkaChannel
a1.channels.c1.kafka.bootstrap.servers = node1:9092,node2:9092
a1.channels.c1.kafka.topic = topic_log01
a1.channels.c1.parseAsFlumeEvent = false
#组装
a1.sources.r1.channels = c1
- 原因就是在创建positionFile的时候父目录已存在
- 场景2:我们生成的日志文件app.log 每经过一天会按照日期重命名文件,然后生成新的app.log,此时flume会重新采集所有的日志信息,导致信息重复采集2次。
- Taildir 说明: Taildir Source 维护了一个 json 格式的 position File,其会定期的往 position File中更新每个文件读取到的最新的位置,因此能够实现断点续传。Position File 的格式如下:
{"inode":2496272,"pos":12,"file":"/opt/module/flume/files/file1.t
xt"}
{"inode":2496275,"pos":12,"file":"/opt/module/flume/files2/log.t
xt"}
- 而flume会同时判断Inode和file来确定是否同一文件
注:Linux 中储存文件元数据的区域就叫做 inode,每个 inode 都有一个号码,操作系统
用 inode 号码来识别不同的文件,Unix/Linux 系统内部不使用文件名,而使用 inode 号码来
识别文件。
3 解决
场景1解决方案有两种:
- 既然是创建父目录已存在,我们可以吧positionFile位置重新配置。
- 修改源代码,我们通过源代码找下处理逻辑,下载1.9.0版本的flume源代码,官网地址:https://archive.apache.org/dist/flume/,找到TailSource 170行
@Override
public synchronized void configure(Context context) {
String fileGroups = context.getString(FILE_GROUPS);
Preconditions.checkState(fileGroups != null, "Missing param: " + FILE_GROUPS);
filePaths = selectByKeys(context.getSubProperties(FILE_GROUPS_PREFIX),
fileGroups.split("\\s+"));
Preconditions.checkState(!filePaths.isEmpty(),
"Mapping for tailing files is empty or invalid: '" + FILE_GROUPS_PREFIX + "'");
String homePath = System.getProperty("user.home").replace('\\', '/');
positionFilePath = context.getString(POSITION_FILE, homePath + DEFAULT_POSITION_FILE);
Path positionFile = Paths.get(positionFilePath);
try {
// 此处创建父目录,如果存在报错
Files.createDirectories(positionFile.getParent());
} catch (IOException e) {
throw new FlumeException("Error creating positionFile parent directories", e);
}
headerTable = getTable(context, HEADERS_PREFIX);
batchSize = context.getInteger(BATCH_SIZE, DEFAULT_BATCH_SIZE);
skipToEnd = context.getBoolean(SKIP_TO_END, DEFAULT_SKIP_TO_END);
byteOffsetHeader = context.getBoolean(BYTE_OFFSET_HEADER, DEFAULT_BYTE_OFFSET_HEADER);
idleTimeout = context.getInteger(IDLE_TIMEOUT, DEFAULT_IDLE_TIMEOUT);
writePosInterval = context.getInteger(WRITE_POS_INTERVAL, DEFAULT_WRITE_POS_INTERVAL);
cachePatternMatching = context.getBoolean(CACHE_PATTERN_MATCHING,
DEFAULT_CACHE_PATTERN_MATCHING);
backoffSleepIncrement = context.getLong(PollableSourceConstants.BACKOFF_SLEEP_INCREMENT,
PollableSourceConstants.DEFAULT_BACKOFF_SLEEP_INCREMENT);
maxBackOffSleepInterval = context.getLong(PollableSourceConstants.MAX_BACKOFF_SLEEP,
PollableSourceConstants.DEFAULT_MAX_BACKOFF_SLEEP);
fileHeader = context.getBoolean(FILENAME_HEADER,
DEFAULT_FILE_HEADER);
fileHeaderKey = context.getString(FILENAME_HEADER_KEY,
DEFAULT_FILENAME_HEADER_KEY);
maxBatchCount = context.getLong(MAX_BATCH_COUNT, DEFAULT_MAX_BATCH_COUNT);
if (maxBatchCount <= 0) {
maxBatchCount = DEFAULT_MAX_BATCH_COUNT;
logger.warn("Invalid maxBatchCount specified, initializing source "
+ "default maxBatchCount of {}", maxBatchCount);
}
if (sourceCounter == null) {
sourceCounter = new SourceCounter(getName());
}
![img](https://img-blog.csdnimg.cn/img_convert/0370582696842b335a3b7bc3b78ca7c2.png)
![img](https://img-blog.csdnimg.cn/img_convert/80377d4b9a3f2132de9fb74330b46cd8.png)
![img](https://img-blog.csdnimg.cn/img_convert/fb9db4ffa197a3007f111e88f93da071.png)
**既有适合小白学习的零基础资料,也有适合3年以上经验的小伙伴深入学习提升的进阶课程,涵盖了95%以上大数据知识点,真正体系化!**
[外链图片转存中...(img-dgz6RJA7-1725714348033)]
[外链图片转存中...(img-jbVaXNwA-1725714348034)]
**既有适合小白学习的零基础资料,也有适合3年以上经验的小伙伴深入学习提升的进阶课程,涵盖了95%以上大数据知识点,真正体系化!**