1. 问题
kafka版本太多,目前最新的已经到2.6.0,但很多线上用的可能还用0.9版本,以往经验,不同版本api存在不少兼容性问题。另外kafka没有实现类似支持es的kibana管理工具,纯靠命令行很蛋疼。
2. 历史
kafka全套解决方案:kafka集群 + kafka-eagle/kafka manager + prometheus + grafana + kafka命令行工具
在上一家公司的时候,kafka+kafka-eagle配合使用挺方便,对kafka-eagle进行二次开发后,能满足大多数需求,能提供多数开发关心的问题,可视化做得不错,简单易用。
前提是kafka集群和kafka-eagle都是自己搭建的,kafka-eagle依赖的版本与kafka集群一致。
3. 现状
现在公司的kafka集群使用的版本是0.9版本,不是我想动就能动,我这边开发只能去匹配集群现有的版本。看kafka-eagle官网,支持0.8-2.x版本,理想很美好,那就试一试。
在自己找寻的kafka测试环境,先创建个topic试试。
kafka-eagle日志报错
2020-08-12 10:50:00 ERROR [Mx4jServiceImpl.org.springframework.scheduling.quartz.SchedulerFactoryBean#1_Worker-10] - JMX service url[10.45.112.79:-1] create has error,msg is null
java.lang.NullPointerException
at org.smartloli.kafka.eagle.core.factory.Mx4jServiceImpl.common(Mx4jServiceImpl.java:228)
at org.smartloli.kafka.eagle.core.factory.Mx4jServiceImpl.replicationBytesInPerSec(Mx4jServiceImpl.java:198)
at org.smartloli.kafka.eagle.web.quartz.MBeanQuartz.kafkaAssembly(MBeanQuartz.java:312)
at org.smartloli.kafka.eagle.web.quartz.MBeanQuartz.kafkaCluster(MBeanQuartz.java:245)
at org.smartloli.kafka.eagle.web.quartz.MBeanQuartz.mbeanQuartz(MBeanQuartz.java:88)
at sun.reflect.GeneratedMethodAccessor74.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.util.MethodInvoker.invoke(MethodInvoker.java:283)
at org.springframework.scheduling.quartz.MethodInvokingJobDetailFactoryBean$MethodInvokingJob.executeInternal(MethodInvokingJobDetailFactoryBean.java:267)
at org.springframework.scheduling.quartz.QuartzJobBean.execute(QuartzJobBean.java:75)
at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
2020-08-12 10:50:15 ERROR [JMXFactoryUtils.pool-758-thread-1] - Block queue is full, error msg is
java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: 10.45.112.79; nested exception is:
java.net.ConnectException: Operation timed out (Connection timed out)]
at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:369)
at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:270)
at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:229)
at org.smartloli.kafka.eagle.common.util.JMXFactoryUtils$1.run(JMXFactoryUtils.java:52)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
kafka日志报错
[2020-08-12 10:49:57,922] ERROR Processor got uncaught exception. (kafka.network.Processor)
java.lang.ArrayIndexOutOfBoundsException: 18
at org.apache.kafka.common.protocol.ApiKeys.forId(ApiKeys.java:68)
at org.apache.kafka.common.requests.AbstractRequest.getRequest(AbstractRequest.java:39)
at kafka.network.RequestChannel$Request.<init>(RequestChannel.scala:79)
at kafka.network.Processor$$anonfun$run$11.apply(SocketServer.scala:426)
at kafka.network.Processor$$anonfun$run$11.apply(SocketServer.scala:421)
at scala.collection.Iterator$class.foreach(Iterator.scala:742)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1194)
at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
at kafka.network.Processor.run(SocketServer.scala:421)
at java.lang.Thread.run(Thread.java:748)
网上找了下kafka服务端类似报错信息, 都指向版本兼容问题。尝试将kafka-eagle降低kafka版本到0.9,编译代码又遇到各种兼容性问题,最后还是放弃。
4. 总结
kafka-eagle的1.x版本就已经开始用kafka 2.0版本,其官网说支持0.8-2.x版本,其实是不严谨的,由于版本兼容问题,最基本的一些操作也无法支持,所以大家使用的时候留意版本兼容问题。