Kafka ERROR Error while deleting the clean shutdown file in dir

背景:kafka 部署在k8s ,其中某个kafka pod 重启后处于异常状态

[2023-02-17 15:44:01,546] ERROR Error while deleting the clean shutdown file in dir /opt/kafka/data/logs (kafka.server.LogDirFailureChannel)
java.nio.file.FileSystemException: /opt/kafka/data/logs/__consumer_offsets-17/leader-epoch-checkpoint: Transport endpoint is not connected
        at sun.nio.fs.UnixException.translateToIOException(UnixException.java:91)
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
        at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:214)
        at java.nio.file.Files.newByteChannel(Files.java:361)
        at java.nio.file.Files.createFile(Files.java:632)
        at kafka.server.checkpoints.CheckpointFile.<init>(CheckpointFile.scala:45)
        at kafka.server.checkpoints.LeaderEpochCheckpointFile.<init>(LeaderEpochCheckpointFile.scala:62)
        at kafka.log.Log.kafka$log$Log$$initializeLeaderEpochCache(Log.scala:302)
        at kafka.log.Log.<init>(Log.scala:232)
        at kafka.log.Log$.apply(Log.scala:1986)
        at kafka.log.LogManager.kafka$log$LogManager$$loadLog(LogManager.scala:265)
        at kafka.log.LogManager$$anonfun$loadLogs$2$$anonfun$11$$anonfun$apply$15$$anonfun$apply$2.apply$mcV$sp(LogManager.scala:345)
        at kafka.utils.CoreUtils$$anon$1.run(CoreUtils.scala:63)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
[2023-02-17 15:44:01,546] ERROR Error while deleting the clean shutdown file in dir /opt/kafka/data/logs (kafka.server.LogDirFailureChannel)
java.io.IOException: Transport endpoint is not connected
        at java.io.UnixFileSystem.createFileExclusively(Native Method)
        at java.io.File.createNewFile(File.java:1012)
        at kafka.log.AbstractIndex.<init>(AbstractIndex.scala:54)
        at kafka.log.OffsetIndex.<init>(OffsetIndex.scala:53)
        at kafka.log.LogSegment$.open(LogSegment.scala:634)
        at kafka.log.Log$$anonfun$kafka$log$Log$$loadSegmentFiles$3.apply(Log.scala:395)
        at kafka.log.Log$$anonfun$kafka$log$Log$$loadSegmentFiles$3.apply(Log.scala:382)
        at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:733)
        at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
        at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186)
        at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:732)
        at kafka.log.Log.kafka$log$Log$$loadSegmentFiles(Log.scala:382)
        at kafka.log.Log$$anonfun$loadSegments$1.apply$mcV$sp(Log.scala:493)
        at kafka.log.Log$$anonfun$loadSegments$1.apply(Log.scala:487)
        at kafka.log.Log$$anonfun$loadSegments$1.apply(Log.scala:487)
        at kafka.log.Log.retryOnOffsetOverflow(Log.scala:1853)
        at kafka.log.Log.loadSegments(Log.scala:487)
        at kafka.log.Log.<init>(Log.scala:237)
        at kafka.log.Log$.apply(Log.scala:1986)
        at kafka.log.LogManager.kafka$log$LogManager$$loadLog(LogManager.scala:265)
        at kafka.log.LogManager$$anonfun$loadLogs$2$$anonfun$11$$anonfun$apply$15$$anonfun$apply$2.apply$mcV$sp(LogManager.scala:345)
        at kafka.utils.CoreUtils$$anon$1.run(CoreUtils.scala:63)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

[root@k8s-master-01 ~]# kubectl describe pv  pvc-1f7e6c8d-ec7d-4735-8623-5e5f5c7a6e86
Name:            pvc-1f7e6c8d-ec7d-4735-8623-5e5f5c7a6e86
Labels:          <none>
Annotations:     Description: Gluster-Internal: Dynamically provisioned PV
                 gluster.kubernetes.io/heketi-volume-id: 101afd6bdb20d6f39c88b56741e3f4f1
                 gluster.org/type: file
                 kubernetes.io/createdby: heketi-dynamic-provisioner
                 pv.beta.kubernetes.io/gid: 2004
                 pv.kubernetes.io/bound-by-controller: yes
                 pv.kubernetes.io/provisioned-by: kubernetes.io/glusterfs
Finalizers:      [kubernetes.io/pv-protection]
StorageClass:    gluster-heketi
Status:          Bound
Claim:           default/datadir-kafka-1
Reclaim Policy:  Delete
Access Modes:    RWO
VolumeMode:      Filesystem
Capacity:        50Gi
Node Affinity:   <none>
Message:
Source:
    Type:                Glusterfs (a Glusterfs mount on the host that shares a pod's lifetime)
    EndpointsName:       glusterfs-dynamic-1f7e6c8d-ec7d-4735-8623-5e5f5c7a6e86
    EndpointsNamespace:  default
    Path:                vol_101afd6bdb20d6f39c88b56741e3f4f1
    ReadOnly:            false
Events:                  <none>

[root@k8s-storage-03 logs]# gluster volume info vol_101afd6bdb20d6f39c88b56741e3f4f1

Volume Name: vol_101afd6bdb20d6f39c88b56741e3f4f1
Type: Distributed-Replicate
Volume ID: ddfff8f4-5369-41ce-94f8-da2105f7fa3f
Status: Started
Snapshot Count: 0
Number of Bricks: 3 x 3 = 9
Transport-type: tcp
Bricks:
Brick1: 10.0.2.44:/var/lib/heketi/mounts/vg_f957b72fcf01d14abe40e4fc44126646/brick_1fba98cf0221789ee59887e30c10ffee/brick
Brick2: 10.0.2.45:/var/lib/heketi/mounts/vg_094f45cfcd0ff5815c3914723876c69e/brick_86551649dc16569a8cd02a5d291db69e/brick
Brick3: 10.0.2.46:/var/lib/heketi/mounts/vg_1f3f18f36dae1a1d968713e583b7972a/brick_46ea9c23f8ce0ff0dff4041d5d1540fd/brick
Brick4: 10.0.2.46:/var/lib/heketi/mounts/vg_1f3f18f36dae1a1d968713e583b7972a/brick_03fb64c2e50d2da7ba0d119db6cdbe6b/brick
Brick5: 10.0.2.44:/var/lib/heketi/mounts/vg_f957b72fcf01d14abe40e4fc44126646/brick_468ca0b1ed787df50165a72305608d25/brick
Brick6: 10.0.2.45:/var/lib/heketi/mounts/vg_094f45cfcd0ff5815c3914723876c69e/brick_284484f3c58fa7d25cf9a2233e8b3cca/brick
Brick7: 10.0.2.44:/var/lib/heketi/mounts/vg_f957b72fcf01d14abe40e4fc44126646/brick_993affb1fcc541b8adb7b674f462cfec/brick
Brick8: 10.0.2.45:/var/lib/heketi/mounts/vg_094f45cfcd0ff5815c3914723876c69e/brick_37a2d4cc20dae75828521ee366162946/brick
Brick9: 10.0.2.46:/var/lib/heketi/mounts/vg_1f3f18f36dae1a1d968713e583b7972a/brick_7a0000e375f3a254f65313f744fa8541/brick
Options Reconfigured:
performance.client-io-threads: off
nfs.disable: on
transport.address-family: inet
user.heketi.id: 101afd6bdb20d6f39c88b56741e3f4f1

解决:删除所有glusterfs brick中的 __consumer_offsets,删除该kafka pod 待其恢复重启

  • 0
    点赞
  • 0
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
"error while closing kafka consumer" 是指在关闭 Kafka 消费者时遇到了错误。 Kafka 是一种分布式流处理平台,用于处理和传输大规模的流式数据。消费者是在 Kafka 集群中读取数据的客户端。关闭消费者时,可能会遇到一些问题。 在关闭 Kafka 消费者时,可能出现以下几种错误: 1. 链接错误: 在关闭消费者时,可能会出现与 Kafka 服务器之间的链接错误。这可能是由于网络问题、服务器故障或不正确的配置导致的。解决此问题的方法是检查网络连接、确保服务器正常运行,并检查消费者的配置是否正确。 2. 资源释放错误: 在关闭消费者时,可能会出现资源释放错误。这可能是由于消费者没有正确释放或关闭使用的资源,从而导致错误。解决此问题的方法是确保在关闭消费者之前,正确释放和关闭使用的资源,如文件句柄、网络连接等。 3. 并发问题: 在多线程或多进程环境下关闭消费者时,可能会出现并发问题。这可能导致资源冲突或竞争条件,从而引发错误。解决此问题的方法是在关闭消费者时使用适当的同步机制,以避免并发问题。 在处理 "error while closing kafka consumer" 错误时,可以采取以下步骤: 1. 检查网络连接是否正常,并确保 Kafka 服务器正常运行。 2. 检查消费者的配置是否正确,包括正确指定 Kafka 服务器的地址、端口以及正确的消费者组等配置。 3. 确保在关闭消费者之前,正确释放和关闭使用的资源,以避免资源泄漏。 4. 在多线程或多进程环境下关闭消费者时,使用适当的同步机制,以避免并发问题。 5. 如果问题仍然存在,可以进一步检查错误日志或调试信息,以找到错误的具体原因,并采取相应的解决措施。 总结起来,处理 "error while closing kafka consumer" 错误需要检查网络连接、验证配置、正确释放资源和处理并发问题等方面,从而确保正确关闭 Kafka 消费者,并避免出现错误。

“相关推荐”对你有帮助么?

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值