解决Kafka本地无法启动问题

问题描述

        本地windows启动kafka：kafka-server-start.bat ../../config/server.properties 时报错，报错如下：

[2023-08-21 14:45:27,408] INFO [GroupCoordinator 0]: Member consumer-groupId01-4-c0bf8cb2-b914-46c0-805a-69f20ee9c498 in group groupId01 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2023-08-21 14:45:27,408] INFO [GroupCoordinator 0]: Member consumer-groupId01-1-b316a1bb-8d39-44c0-bb7c-bbf903ad7cc1 in group groupId01 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2023-08-21 14:45:27,409] INFO [GroupCoordinator 0]: Group groupId01 with generation 41 is now empty (__consumer_offsets-1) (kafka.coordinator.group.GroupCoordinator)
[2023-08-21 14:45:27,411] INFO [LocalLog partition=__consumer_offsets-1, dir=E:\kafka\kafka_2.13-3.2.3\tmp\kafka-logs] Rolled new log segment at offset 72 in 2 ms. (kafka.log.LocalLog)
[2023-08-21 14:45:30,910] ERROR Failed to clean up log for __consumer_offsets-1 in dir E:\kafka\kafka_2.13-3.2.3\tmp\kafka-logs due to IOException (kafka.server.LogDirFailureChannel)
java.nio.file.FileSystemException: E:\kafka\kafka_2.13-3.2.3\tmp\kafka-logs\__consumer_offsets-1\00000000000000000000.timeindex.cleaned -> E:\kafka\kafka_2.13-3.2.3\tmp\kafka-logs\__consumer_offsets-1\00000000000000000000.timeindex.swap: 另一个程序正在使用此文件，进程无法访问。

        at sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:86)
        at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97)
        at sun.nio.fs.WindowsFileCopy.move(WindowsFileCopy.java:387)
        at sun.nio.fs.WindowsFileSystemProvider.move(WindowsFileSystemProvider.java:287)
        at java.nio.file.Files.move(Files.java:1395)
        at org.apache.kafka.common.utils.Utils.atomicMoveWithFallback(Utils.java:935)
        at kafka.log.AbstractIndex.renameTo(AbstractIndex.scala:210)
        at kafka.log.LazyIndex$IndexValue.renameTo(LazyIndex.scala:155)
        at kafka.log.LazyIndex.$anonfun$renameTo$1(LazyIndex.scala:79)
        at kafka.log.LazyIndex.renameTo(LazyIndex.scala:79)
        at kafka.log.LogSegment.changeFileSuffixes(LogSegment.scala:493)
        at kafka.log.LocalLog$.$anonfun$replaceSegments$4(LocalLog.scala:890)
        at kafka.log.LocalLog$.$anonfun$replaceSegments$4$adapted(LocalLog.scala:890)
        at scala.collection.immutable.List.foreach(List.scala:333)
        at kafka.log.LocalLog$.replaceSegments(LocalLog.scala:890)
        at kafka.log.UnifiedLog.replaceSegments(UnifiedLog.scala:1949)
        at kafka.log.Cleaner.cleanSegments(LogCleaner.scala:617)
        at kafka.log.Cleaner.$anonfun$doClean$6(LogCleaner.scala:537)
        at kafka.log.Cleaner.$anonfun$doClean$6$adapted(LogCleaner.scala:536)
        at scala.collection.immutable.List.foreach(List.scala:333)
        at kafka.log.Cleaner.doClean(LogCleaner.scala:536)
        at kafka.log.Cleaner.clean(LogCleaner.scala:501)
        at kafka.log.LogCleaner$CleanerThread.cleanLog(LogCleaner.scala:385)
        at kafka.log.LogCleaner$CleanerThread.cleanFilthiestLog(LogCleaner.scala:357)
        at kafka.log.LogCleaner$CleanerThread.tryCleanFilthiestLog(LogCleaner.scala:336)
        at kafka.log.LogCleaner$CleanerThread.doWork(LogCleaner.scala:323)
        at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:96)
        Suppressed: java.nio.file.FileSystemException: E:\kafka\kafka_2.13-3.2.3\tmp\kafka-logs\__consumer_offsets-1\00000000000000000000.timeindex.cleaned -> E:\kafka\kafka_2.13-3.2.3\tmp\kafka-logs\__consumer_offsets-1\00000000000000000000.timeindex.swap: 另一个程序正在使用此文件，进程无法访问。

                at sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:86)
                at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97)
                at sun.nio.fs.WindowsFileCopy.move(WindowsFileCopy.java:301)
                at sun.nio.fs.WindowsFileSystemProvider.move(WindowsFileSystemProvider.java:287)
                at java.nio.file.Files.move(Files.java:1395)
                at org.apache.kafka.common.utils.Utils.atomicMoveWithFallback(Utils.java:932)
                ... 21 more
[2023-08-21 14:45:30,916] WARN [ReplicaManager broker=0] Stopping serving replicas in dir E:\kafka\kafka_2.13-3.2.3\tmp\kafka-logs (kafka.server.ReplicaManager)
[2023-08-21 14:45:30,925] INFO [ReplicaFetcherManager on broker 0] Removed fetcher for partitions HashSet(__consumer_offsets-22, __consumer_offsets-25, demo3-0, __consumer_offsets-13, __consumer_offsets-30, demo1-0, __consumer_offsets-8, __consumer_offsets-21, __consumer_offsets-4, __consumer_offsets-27, __consumer_offsets-7, __consumer_offsets-9, __consumer_offsets-46, __consumer_offsets-35, demo2-0, __consumer_offsets-41, __consumer_offsets-33, __consumer_offsets-23, __consumer_offsets-49, __consumer_offsets-47, __consumer_offsets-16, __consumer_offsets-28, demo5-0, __consumer_offsets-31, __consumer_offsets-36, __consumer_offsets-42, __consumer_offsets-3, __consumer_offsets-18, __consumer_offsets-37, demo-0, __consumer_offsets-15, __consumer_offsets-24, demo4-0, __consumer_offsets-38, __consumer_offsets-17, __consumer_offsets-48, __consumer_offsets-19, __consumer_offsets-11, __consumer_offsets-2, __consumer_offsets-43, __consumer_offsets-6, __consumer_offsets-14, __consumer_offsets-20, __consumer_offsets-0, __consumer_offsets-44, __consumer_offsets-39, __consumer_offsets-12, __consumer_offsets-45, __consumer_offsets-1, __consumer_offsets-5, __consumer_offsets-26, __consumer_offsets-29, __consumer_offsets-34, __consumer_offsets-10, __consumer_offsets-32, __consumer_offsets-40) (kafka.server.ReplicaFetcherManager)
[2023-08-21 14:45:30,926] INFO [ReplicaAlterLogDirsManager on broker 0] Removed fetcher for partitions HashSet(__consumer_offsets-22, __consumer_offsets-25, demo3-0, __consumer_offsets-13, __consumer_offsets-30, demo1-0, __consumer_offsets-8, __consumer_offsets-21, __consumer_offsets-4, __consumer_offsets-27, __consumer_offsets-7, __consumer_offsets-9, __consumer_offsets-46, __consumer_offsets-35, demo2-0, __consumer_offsets-41, __consumer_offsets-33, __consumer_offsets-23, __consumer_offsets-49, __consumer_offsets-47, __consumer_offsets-16, __consumer_offsets-28, demo5-0, __consumer_offsets-31, __consumer_offsets-36, __consumer_offsets-42, __consumer_offsets-3, __consumer_offsets-18, __consumer_offsets-37, demo-0, __consumer_offsets-15, __consumer_offsets-24, demo4-0, __consumer_offsets-38, __consumer_offsets-17, __consumer_offsets-48, __consumer_offsets-19, __consumer_offsets-11, __consumer_offsets-2, __consumer_offsets-43, __consumer_offsets-6, __consumer_offsets-14, __consumer_offsets-20, __consumer_offsets-0, __consumer_offsets-44, __consumer_offsets-39, __consumer_offsets-12, __consumer_offsets-45, __consumer_offsets-1, __consumer_offsets-5, __consumer_offsets-26, __consumer_offsets-29, __consumer_offsets-34, __consumer_offsets-10, __consumer_offsets-32, __consumer_offsets-40) (kafka.server.ReplicaAlterLogDirsManager)
[2023-08-21 14:45:30,974] WARN [ReplicaManager broker=0] Broker 0 stopped fetcher for partitions __consumer_offsets-22,__consumer_offsets-25,demo3-0,__consumer_offsets-13,__consumer_offsets-30,demo1-0,__consumer_offsets-8,__consumer_offsets-21,__consumer_offsets-4,__consumer_offsets-27,__consumer_offsets-7,__consumer_offsets-9,__consumer_offsets-46,__consumer_offsets-35,demo2-0,__consumer_offsets-41,__consumer_offsets-33,__consumer_offsets-23,__consumer_offsets-49,__consumer_offsets-47,__consumer_offsets-16,__consumer_offsets-28,demo5-0,__consumer_offsets-31,__consumer_offsets-36,__consumer_offsets-42,__consumer_offsets-3,__consumer_offsets-18,__consumer_offsets-37,demo-0,__consumer_offsets-15,__consumer_offsets-24,demo4-0,__consumer_offsets-38,__consumer_offsets-17,__consumer_offsets-48,__consumer_offsets-19,__consumer_offsets-11,__consumer_offsets-2,__consumer_offsets-43,__consumer_offsets-6,__consumer_offsets-14,__consumer_offsets-20,__consumer_offsets-0,__consumer_offsets-44,__consumer_offsets-39,__consumer_offsets-12,__consumer_offsets-45,__consumer_offsets-1,__consumer_offsets-5,__consumer_offsets-26,__consumer_offsets-29,__consumer_offsets-34,__consumer_offsets-10,__consumer_offsets-32,__consumer_offsets-40 and stopped moving logs for partitions  because they are in the failed log directory E:\kafka\kafka_2.13-3.2.3\tmp\kafka-logs. (kafka.server.ReplicaManager)
[2023-08-21 14:45:30,974] WARN Stopping serving logs in dir E:\kafka\kafka_2.13-3.2.3\tmp\kafka-logs (kafka.log.LogManager)
[2023-08-21 14:45:30,976] ERROR Failed to clean up log for __consumer_offsets-1 in dir E:\kafka\kafka_2.13-3.2.3\tmp\kafka-logs due to IOException (kafka.server.LogDirFailureChannel)
java.nio.file.FileSystemException: E:\kafka\kafka_2.13-3.2.3\tmp\kafka-logs\__consumer_offsets-1\00000000000000000000.timeindex.cleaned: 另一个程序正在使用此文件，进程无法访问。

        at sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:86)
        at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97)
        at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:102)
        at sun.nio.fs.WindowsFileSystemProvider.implDelete(WindowsFileSystemProvider.java:269)
        at sun.nio.fs.AbstractFileSystemProvider.deleteIfExists(AbstractFileSystemProvider.java:108)
        at java.nio.file.Files.deleteIfExists(Files.java:1165)
        at kafka.log.UnifiedLog$.deleteFileIfExists(UnifiedLog.scala:1837)
        at kafka.log.LogSegment$.deleteIfExists(LogSegment.scala:683)
        at kafka.log.LocalLog$.createNewCleanedSegment(LocalLog.scala:973)
        at kafka.log.Cleaner.cleanSegments(LogCleaner.scala:567)
        at kafka.log.Cleaner.$anonfun$doClean$6(LogCleaner.scala:537)
        at kafka.log.Cleaner.$anonfun$doClean$6$adapted(LogCleaner.scala:536)
        at scala.collection.immutable.List.foreach(List.scala:333)
        at kafka.log.Cleaner.doClean(LogCleaner.scala:536)
        at kafka.log.Cleaner.clean(LogCleaner.scala:501)
        at kafka.log.LogCleaner$CleanerThread.cleanLog(LogCleaner.scala:385)
        at kafka.log.LogCleaner$CleanerThread.cleanFilthiestLog(LogCleaner.scala:357)
        at kafka.log.LogCleaner$CleanerThread.tryCleanFilthiestLog(LogCleaner.scala:336)
        at kafka.log.LogCleaner$CleanerThread.doWork(LogCleaner.scala:323)
        at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:96)
[2023-08-21 14:45:30,981] ERROR Shutdown broker because all log dirs in E:\kafka\kafka_2.13-3.2.3\tmp\kafka-logs have failed (kafka.log.LogManager)

E:\kafka\kafka_2.13-3.2.3\bin\windows>kafka-server-start.bat ../../config/server.properties
[2023-08-21 14:46:05,414] INFO Registered kafka:type=kafka.Log4jController MBean (kafka.utils.Log4jControllerRegistration$)
[2023-08-21 14:46:05,798] INFO Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation (org.apache.zookeeper.common.X509Util)
[2023-08-21 14:46:05,888] INFO starting (kafka.server.KafkaServer)
[2023-08-21 14:46:05,889] INFO Connecting to zookeeper on localhost:2181 (kafka.server.KafkaServer)
[2023-08-21 14:46:05,904] INFO [ZooKeeperClient Kafka server] Initializing a new session to localhost:2181. (kafka.zookeeper.ZooKeeperClient)
[2023-08-21 14:46:05,909] INFO Client environment:zookeeper.version=3.6.3--6401e4ad2087061bc6b9f80dec2d69f2e3c8660a, built on 04/08/2021 16:35 GMT (org.apache.zookeeper.ZooKeeper)
[2023-08-21 14:46:05,909] INFO Client environment:host.name=DESKTOP-IR45FRI (org.apache.zookeeper.ZooKeeper)
[2023-08-21 14:46:05,909] INFO Client environment:java.version=1.8.0_151 (org.apache.zookeeper.ZooKeeper)
[2023-08-21 14:46:05,909] INFO Client environment:java.vendor=Oracle Corporation (org.apache.zookeeper.ZooKeeper)
[2023-08-21 14:46:05,910] INFO Client environment:java.home=C:\Program Files\Java\jdk1.8.0_151\jre (org.apache.zookeeper.ZooKeeper)

解决方案

        报错信息提示kafka-logs\__consumer_offsets-1\00000000000000000000.timeindex.swap文件被另一个程序使用，进程无法访问，奇怪！此文件只有kafka才会使用，所以会不会是之前启动后未正常关闭kafka导致的，既然跟文件有关，索性就把tmp文件夹下的kafka-logs文件夹删除了，再重新启动后正常。

        亦或者是更改kafka日志的存放路径（建议此方法！！！）。

        编辑E:\kafka\kafka_2.13-3.2.3\config\server.properties，修改路径即可。