windows版kafka异常java.nio.file.FileSystemException另一个程序正在使用此文件，进程无法访问

最新推荐文章于 2024-06-14 16:45:18 发布

山水牧羊

最新推荐文章于 2024-06-14 16:45:18 发布

阅读量3.1k

点赞数 5

分类专栏：读书笔记文章标签： Kafka 微服务

本文链接：https://blog.csdn.net/yeqiyugood/article/details/117477562

版权

读书笔记专栏收录该内容

6 篇文章 0 订阅

订阅专栏

环境

Windows2012 R2

kafka_2.12-2.3.0

现象

[2021-06-01 18:40:38,102] INFO [Log partition=testMcdull222-0, dir=D:\DEV\MQ\kafka_2.12-2.3.0\kafka_2.12-2.3.0\logs] Found deletable segments with base offsets [0] due to retention time 2000ms breach (kafka.log.Log)
[2021-06-01 18:40:38,111] INFO [Log partition=testMcdull222-0, dir=D:\DEV\MQ\kafka_2.12-2.3.0\kafka_2.12-2.3.0\logs] Rolled new log segment at offset 3064 in 7 ms. (kafka.log.Log)
[2021-06-01 18:40:38,112] INFO [Log partition=testMcdull222-0, dir=D:\DEV\MQ\kafka_2.12-2.3.0\kafka_2.12-2.3.0\logs] Scheduling log segment [baseOffset 0, size 220487] for deletion. (kafka.log.Log)
[2021-06-01 18:40:38,120] ERROR Error while deleting segments for testMcdull222-0 in dir D:\DEV\MQ\kafka_2.12-2.3.0\kafka_2.12-2.3.0\logs (kafka.server.LogDirFailureChannel)
java.nio.file.FileSystemException: D:\DEV\MQ\kafka_2.12-2.3.0\kafka_2.12-2.3.0\logs\testMcdull222-0\00000000000000000000.index -> D:\DEV\MQ\kafka_2.12-2.3.0\kafka_2.12-2.3.0\logs\testMcdull222-0\00000000000000000000.index.deleted: Á풻¸ö³̐򕽔ڊ¹Ӄ´ˎļþ£¬½ø³̎޷¨·Îʡ£

	at sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:86)
	at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97)
	at sun.nio.fs.WindowsFileCopy.move(WindowsFileCopy.java:387)
	at sun.nio.fs.WindowsFileSystemProvider.move(WindowsFileSystemProvider.java:287)
	at java.nio.file.Files.move(Files.java:1395)
	at org.apache.kafka.common.utils.Utils.atomicMoveWithFallback(Utils.java:815)
	at kafka.log.AbstractIndex.renameTo(AbstractIndex.scala:209)
	at kafka.log.LogSegment.changeFileSuffixes(LogSegment.scala:509)
	at kafka.log.Log.asyncDeleteSegment(Log.scala:1962)
	at kafka.log.Log.deleteSegment(Log.scala:1947)
	at kafka.log.Log.$anonfun$deleteSegments$3(Log.scala:1493)
	at kafka.log.Log.$anonfun$deleteSegments$3$adapted(Log.scala:1493)
	at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
	at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
	at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
	at kafka.log.Log.$anonfun$deleteSegments$2(Log.scala:1493)
	at scala.runtime.java8.JFunction0$mcI$sp.apply(JFunction0$mcI$sp.java:23)
	at kafka.log.Log.maybeHandleIOException(Log.scala:2065)
	at kafka.log.Log.deleteSegments(Log.scala:1484)
	at kafka.log.Log.deleteOldSegments(Log.scala:1479)
	at kafka.log.Log.deleteRetentionMsBreachedSegments(Log.scala:1557)
	at kafka.log.Log.deleteOldSegments(Log.scala:1547)
	at kafka.log.LogManager.$anonfun$cleanupLogs$3(LogManager.scala:914)
	at kafka.log.LogManager.$anonfun$cleanupLogs$3$adapted(LogManager.scala:911)
	at scala.collection.immutable.List.foreach(List.scala:392)
	at kafka.log.LogManager.cleanupLogs(LogManager.scala:911)
	at kafka.log.LogManager.$anonfun$startup$2(LogManager.scala:395)
	at kafka.utils.KafkaScheduler.$anonfun$schedule$2(KafkaScheduler.scala:114)
	at kafka.utils.CoreUtils$$anon$1.run(CoreUtils.scala:65)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
	Suppressed: java.nio.file.FileSystemException: D:\DEV\MQ\kafka_2.12-2.3.0\kafka_2.12-2.3.0\logs\testMcdull222-0\00000000000000000000.index -> D:\DEV\MQ\kafka_2.12-2.3.0\kafka_2.12-2.3.0\logs\testMcdull222-0\00000000000000000000.index.deleted: Á풻¸ö³̐򕽔ڊ¹Ӄ´ˎļþ£¬½ø³̎޷¨·Îʡ£

		at sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:86)
		at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97)
		at sun.nio.fs.WindowsFileCopy.move(WindowsFileCopy.java:301)
		at sun.nio.fs.WindowsFileSystemProvider.move(WindowsFileSystemProvider.java:287)
		at java.nio.file.Files.move(Files.java:1395)
		at org.apache.kafka.common.utils.Utils.atomicMoveWithFallback(Utils.java:812)
		... 30 more

原因

kafka配置文件server.properties中，log.retention.hours=168

当kafka运行168小时后，将会删除保存消息的文件 logs\testMcdull222-0\00000000000000000000.index但是，该文件已被kafka打开，无法完成删除。

确切讲，是在windows下无法删除。因为Windows文件系统的锁机制，所以该函数不能在windows下正常运行。

解决方法

1.调整操作系统，将Windows改为Linux。

在Windows安装虚拟机，运行Linux。
在Windows运行Docker容器，运行Kafka。
在Windows运行WSL。

绕过错误，备选方案。

2.修改配置文件，永久不清理数据文件

会造成文件体积较大，备选方案。

log.retention.hours=-1
log.cleaner.enable=false

3.手工关闭Kafka，然后删除文件。

会造成服务中断，不应采纳。

4.更新kafka补丁

比较合理的方法，谁有错误谁修正。

具体错误修正方法

https://github.com/apache/kafka/pull/6329/commits/3eceb9ea1d96d545d1c60713ea40e1ce7b354dce

重新编译Kafka

编译方法_gradle

https://github.com/apache/kafka/tree/0baf9c158b5681a55df4de3a0e6193d32b1433ff

问题修正后删除文件日志

[2021-06-01 19:18:25,481] INFO [Log partition=testMcdull222-0, dir=D:\DEV\MQ\kafka_2.12-2.3.0\kafka_2.12-2.3.0\logs] Deleting segment 23158 (kafka.log.Log)
[2021-06-01 19:18:25,482] INFO Deleted log D:\DEV\MQ\kafka_2.12-2.3.0\kafka_2.12-2.3.0\logs\testMcdull222-0\00000000000000023158.log.deleted. (kafka.log.LogSegment)
[2021-06-01 19:18:25,489] INFO Deleted offset index D:\DEV\MQ\kafka_2.12-2.3.0\kafka_2.12-2.3.0\logs\testMcdull222-0\00000000000000023158.index.deleted. (kafka.log.LogSegment)
[2021-06-01 19:18:25,494] INFO Deleted time index D:\DEV\MQ\kafka_2.12-2.3.0\kafka_2.12-2.3.0\logs\testMcdull222-0\00000000000000023158.timeindex.deleted. (kafka.log.LogSegment)
[2021-06-01 19:18:25,494] INFO [Log partition=testMcdull222-1, dir=D:\DEV\MQ\kafka_2.12-2.3.0\kafka_2.12-2.3.0\logs] Deleting segment 23157 (kafka.log.Log)
[2021-06-01 19:18:25,495] INFO [Log partition=testMcdull222-2, dir=D:\DEV\MQ\kafka_2.12-2.3.0\kafka_2.12-2.3.0\logs] Deleting segment 3064 (kafka.log.Log)
[2021-06-01 19:18:25,495] INFO Deleted log D:\DEV\MQ\kafka_2.12-2.3.0\kafka_2.12-2.3.0\logs\testMcdull222-1\00000000000000023157.log.deleted. (kafka.log.LogSegment)
[2021-06-01 19:18:25,496] INFO Deleted log D:\DEV\MQ\kafka_2.12-2.3.0\kafka_2.12-2.3.0\logs\testMcdull222-2\00000000000000003064.log.deleted. (kafka.log.LogSegment)
[2021-06-01 19:18:25,502] INFO Deleted offset index D:\DEV\MQ\kafka_2.12-2.3.0\kafka_2.12-2.3.0\logs\testMcdull222-1\00000000000000023157.index.deleted. (kafka.log.LogSegment)
[2021-06-01 19:18:25,502] INFO Deleted offset index D:\DEV\MQ\kafka_2.12-2.3.0\kafka_2.12-2.3.0\logs\testMcdull222-2\00000000000000003064.index.deleted. (kafka.log.LogSegment)
[2021-06-01 19:18:25,507] INFO Deleted time index D:\DEV\MQ\kafka_2.12-2.3.0\kafka_2.12-2.3.0\logs\testMcdull222-1\00000000000000023157.timeindex.deleted. (kafka.log.LogSegment)
[2021-06-01 19:18:25,507] INFO Deleted time index D:\DEV\MQ\kafka_2.12-2.3.0\kafka_2.12-2.3.0\logs\testMcdull222-2\00000000000000003064.timeindex.deleted. (kafka.log.LogSegment)