书接上文,继续讲可能遇到的错误
5.Brokers drop out of ISR
可能原因:
- kafka默认的是大于4000message滞后的时候就drop ISR
- replica.lag.max.messages =4000,如果延时太多,kafka认为网络状况不好,就不要这个follower了,所以当网络状况不好,broker较少,或者机子的吞吐量不够的时候,可以提高这个默认的值
- 如果replia.fetch.max.bytes小于message.max.bytes(broker能接受的最大的bytes),message can not be replicated。replica也想consumer一样,不停的从leader那获得data,所以,replica们有自己的能接受的最大值byte
6. partition中的消息不balance
- 这个就是个random的问题了,不指定key的话,kafka自己产生random 数字来balance,它自己选择一个partition然后发送一段时间,再换partition发,不想这样,可以自己产生random的key给kafka。或者改topic.metadata.refresh.interval.
- 也可以自己写一个partitioner
7. consumers/brokers are running out of memory
- 要知道需要多少内存,首先需要知道我们要多少个replica,然后看看每个replica要max多少bytes。具体是replica.message.max.bytes. fetch.message.max.bytes
8. 昨天遇到一个新问题,还没解决,先贴出来。等解决了回来告诉大家。
-- FATAL [Kafka Server 0], Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
kafka.common.KafkaException: Failed to acquire lock on file .lock in /tmp/kafka-logs. A Kafka instance in another process or thread is using this directory.
at kafka.log.LogManager$$anonfun$lockLogDirs$1.apply(LogManager.scala:98)
at kafka.log.LogManager$$anonfun$lockLogDirs$1.apply(LogManager.scala:95)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:34)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
at scala.collection.AbstractTraversable.map(Traversable.scala:105)
at kafka.log.LogManager.lockLogDirs(LogManager.scala:95)
at kafka.log.LogManager.<init>(LogManager.scala:57)
at kafka.server.KafkaServer.createLogManager(KafkaServer.scala:335)
at kafka.server.KafkaServer.startup(KafkaServer.scala:85)
at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:29)
at kafka.Kafka$.main(Kafka.scala:46)
at kafka.Kafka.main(Kafka.scala)
kafka.common.KafkaException: Failed to acquire lock on file .lock in /tmp/kafka-logs. A Kafka instance in another process or thread is using this directory.
at kafka.log.LogManager$$anonfun$lockLogDirs$1.apply(LogManager.scala:98)
at kafka.log.LogManager$$anonfun$lockLogDirs$1.apply(LogManager.scala:95)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:34)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
at scala.collection.AbstractTraversable.map(Traversable.scala:105)
at kafka.log.LogManager.lockLogDirs(LogManager.scala:95)
at kafka.log.LogManager.<init>(LogManager.scala:57)
at kafka.server.KafkaServer.createLogManager(KafkaServer.scala:335)
at kafka.server.KafkaServer.startup(KafkaServer.scala:85)
at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:29)
at kafka.Kafka$.main(Kafka.scala:46)
at kafka.Kafka.main(Kafka.scala)
对了,kafka有个mailing list,有不会的可以去求助。
Q-day80