kafka运维填坑

转载自：https://www.jianshu.com/p/d2cbaae38014

• 前提: 只针对Kafka 0.9.0.1版本;
• 说是运维,其实偏重于问题解决;
• 大部分解决方案都是google而来, 我只是作了次搬运工;
• 有些问题的解决方案未必一定是通用的, 若应用到线上请慎重;
• 如有疏漏之处, 欢迎大家批评指正;
• 列表:
  1. Replica无法从leader同步消息
  2. Broker到zk集群的连接不时会断开重断
  3. Broker重启耗时很久
  4. 不允许脏主选举导致Broker被强制关闭
  5. Replica从错误的Partition leader上去同步数据
  6. __consumer_offsets日志无法被清除
  7. GC问题
  8. zk和kafka部署
  9. 监控很重要
  10. 大量异常: Attempted to decrease connection count for address with no connections
  11. 新版sdk访问较旧版的kafka, 发送kafka不支持的request
  12. 频繁FullGC
  13. 机器Swap使用

---

Replica无法从leader同步消息

• 现象: 集群上某topic原来只有单复本, 增加双复本后,发现有些partition没有从leader同步数据,导致isr列表中一直没有新增的replica;
• 日志分析:

[2017-09-20 19:37:05,265] ERROR Found invalid messages during fetch for partition [xxxx,87] offset 1503297 error Message is corrupt (stored crc = 286782282, computed crc = 400317671) (kafka.server.ReplicaFetcherThread) [2017-09-20 19:37:05,458] ERROR Found invalid messages during fetch for partition [xxxx,75] offset 1501373 error Message found with corrupt size (0) in shallow iterator (kafka.server.ReplicaFetcherThread) [2017-09-20 19:37:07,455] ERROR [ReplicaFetcherThread-0-5], Error due to (kafka.server.ReplicaFetcherThread) kafka.common.KafkaException: error processing data for partition [xxxx,87] offset 1503346 at kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$2$$anonfun$apply$mcV$sp$1$$anonfun$apply$2.apply(AbstractFetcherThread.scala:147) at kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$2$$anonfun$apply$mcV$sp$1$$anonfun$apply$2.apply(AbstractFetcherThread.scala:122) at scala.Option.foreach(Option.scala:257) at kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$2$$anonfun$apply$mcV$sp$1.apply(AbstractFetcherThread.scala:122) at kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$2$$anonfun$apply$mcV$sp$1.apply(AbstractFetcherThread.scala:120) at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99) at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99) at scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:230) at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:40) at scala.collection.mutable.HashMap.foreach(

转载于:https://www.cnblogs.com/lushilin/p/11545001.html