JMX监控kafka各种指标

nohup KAFKA_JMX_OPTS="-Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname=192.168.71.25 -Djava.net.preferIPv4Stack=true -Dcom.sun.management.jmxremote.port=9999" ./kafka-server-start.sh …/config/server.properties 1>/dev/null 2>&1 &

Kafka常用JMX监控指标整理【实战笔记】
目录
一、系统相关指标
二、GC相关指标
三、JVM相关指标
四、Topic相关指标
五、Broker相关指标
六、系列文章

一、系统相关指标
1.系统信息收集
java.lang:type=OperatingSystem

{“freePhysicalMemorySize”:“806023168”,“maxFileDescriptorCount”:“4096”,“openFileDescriptorCount”:“283”,“processCpuLoad”:“0.0017562901839817224”,“systemCpuLoad”:“0.014336627412954635”,“systemLoadAverage”:“0.37”}
2.Thread信息收集
java.lang:type=Threading

{“peakThreadCount”:“88”,“threadCount”:“74”}
3.获取mmaped和direct空间

通过BufferPoolMXBean获取used、capacity、count

二、GC相关指标
1.Young GC
java.lang:type=GarbageCollector,name=G1 Young Generation

{“collectionCount”:“534”,“collectionTime”:“8258”}
2.Old GC
java.lang:type=GarbageCollector,name=G1 Old Generation

{“collectionCount”:“0”,“collectionTime”:“0”}

三、JVM相关指标
通过MemoryMXBean获取JVM相关信息HeapMemoryUsage和NonHeapMemoryUsage；通过MemoryPoolMXBean获取其他JVM内存空间指标，例如：Metaspace、Codespace等

四、Topic相关指标
1.Topic消息入站速率（Byte）
kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec,topic=" + topic

{“count”:“0”,“fifteenMinuteRate”:“0.0”,“fiveMinuteRate”:“0.0”,“meanRate”:“0.0”,“oneMinuteRate”:“0.0”}
2.Topic消息出站速率（Byte）
kafka.server:type=BrokerTopicMetrics,name=BytesOutPerSec,topic=" + topic

{“count”:“0”,“fifteenMinuteRate”:“0.0”,“fiveMinuteRate”:“0.0”,“meanRate”:“0.0”,“oneMinuteRate”:“0.0”}
3.Topic请求被拒速率
kafka.server:type=BrokerTopicMetrics,name=BytesRejectedPerSec,topic=" + topic

{“count”:“0”,“fifteenMinuteRate”:“0.0”,“fiveMinuteRate”:“0.0”,“meanRate”:“0.0”,“oneMinuteRate”:“0.0”}
4.Topic失败拉去请求速率
kafka.server:type=BrokerTopicMetrics,name=FailedFetchRequestsPerSec,topic=" + topic;

{“count”:“0”,“fifteenMinuteRate”:“0.0”,“fiveMinuteRate”:“0.0”,“meanRate”:“0.0”,“oneMinuteRate”:“0.0”}
5.Topic发送请求失败速率
kafka.server:type=BrokerTopicMetrics,name=FailedProduceRequestsPerSec,topic=" + topic

{“count”:“0”,“fifteenMinuteRate”:“0.0”,“fiveMinuteRate”:“0.0”,“meanRate”:“0.0”,“oneMinuteRate”:“0.0”}
6.Topic消息入站速率（message）
kafka.server:type=BrokerTopicMetrics,name=MessagesInPerSec,topic=" + topic
{“count”:“0”,“fifteenMinuteRate”:“0.0”,“fiveMinuteRate”:“0.0”,“meanRate”:“0.0”,“oneMinuteRate”:“0.0”}

五、Broker相关指标
1.Log flush rate and time
kafka.log:type=LogFlushStats,name=LogFlushRateAndTimeMs

{“50thPercentile”:“1.074103”,“75thPercentile”:“1.669793”,“95thPercentile”:“6.846556”,“98thPercentile”:“6.846556”,“999thPercentile”:“6.846556”,“99thPercentile”:“6.846556”,“count”:“19”,“max”:“6.846556”,“mean”:“1.628646052631579”,“min”:“0.512879”,“stdDev”:“1.6007003364105892”}
2.同步失效的副本数
kafka.server:type=ReplicaManager,name=UnderReplicatedPartitions

{“value”:“0”}
3.消息入站速率（消息数）
kafka.server:type=BrokerTopicMetrics,name=MessagesInPerSec

{“count”:“86845”,“fifteenMinuteRate”:“0.6456600497006455”,“fiveMinuteRate”:“0.6444164288097876”,“meanRate”:“0.5314899330400695”,“oneMinuteRate”:“0.6494649408329609”}
4.消息入站速率（Byte）
kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec

{“count”:“57302357”,“fifteenMinuteRate”:“379.11342092748146”,“fiveMinuteRate”:“371.8482236385939”,“meanRate”:“351.37122686037435”,“oneMinuteRate”:“351.8348952308101”}
5.消息出站速率（Byte）
kafka.server:type=BrokerTopicMetrics,name=BytesOutPerSec

{“count”:“246”,“fifteenMinuteRate”:“4.508738367219028E-34”,“fiveMinuteRate”:“1.4721921790135324E-98”,“meanRate”:“0.0015031168286836175”,“oneMinuteRate”:“2.964393875E-314”}
6.请求被拒速率
kafka.server:type=BrokerTopicMetrics,name=BytesRejectedPerSec

{“count”:“0”,“fifteenMinuteRate”:“0.0”,“fiveMinuteRate”:“0.0”,“meanRate”:“0.0”,“oneMinuteRate”:“0.0”}
7.失败拉去请求速率
kafka.server:type=BrokerTopicMetrics,name=FailedFetchRequestsPerSec

{“count”:“0”,“fifteenMinuteRate”:“0.0”,“fiveMinuteRate”:“0.0”,“meanRate”:“0.0”,“oneMinuteRate”:“0.0”}
8.发送请求失败速率
kafka.server:type=BrokerTopicMetrics,name=FailedProduceRequestsPerSec

{“count”:“0”,“fifteenMinuteRate”:“0.0”,“fiveMinuteRate”:“0.0”,“meanRate”:“0.0”,“oneMinuteRate”:“0.0”}
9.Leader副本数
kafka.server:type=ReplicaManager,name=LeaderCount

{“value”:“92”}
10.Partition数量
kafka.server:type=ReplicaManager,name=PartitionCount

{“value”:“135”}
11.下线Partition数量
kafka.controller:type=KafkaController,name=OfflinePartitionsCount

{“value”:“0”}
12.Broker网络处理线程空闲率
kafka.server:type=KafkaRequestHandlerPool,name=RequestHandlerAvgIdlePercent

{“count”:“164506926671008”,“fifteenMinuteRate”:“0.9999327359820058”,“fiveMinuteRate”:“1.0000290054537715”,“meanRate”:“0.9998854371393514”,“oneMinuteRate”:“1.0007836499581673”}
13.Leader选举比率
kafka.controller:type=ControllerStats,name=LeaderElectionRateAndTimeMs

{“count”:“7”,“fifteenMinuteRate”:“5.134993718576819E-82”,“fiveMinuteRate”:“6.882658450509451E-240”,“meanRate”:“4.2525243043608314E-5”,“oneMinuteRate”:“2.964393875E-314”}
14.Unclean Leader选举比率
kafka.controller:type=ControllerStats,name=UncleanLeaderElectionsPerSec

{“count”:“0”,“fifteenMinuteRate”:“0.0”,“fiveMinuteRate”:“0.0”,“meanRate”:“0.0”,“oneMinuteRate”:“0.0”}
15.Controller存活数量
kafka.controller:type=KafkaController,name=ActiveControllerCount

{“value”:“1”}
16.请求速率
kafka.network:type=RequestMetrics,name=RequestsPerSec,request=Produce

{“count”:“83233”,“fifteenMinuteRate”:“0.6303485369828705”,“fiveMinuteRate”:“0.6357199085092445”,“meanRate”:“0.5046486472186744”,“oneMinuteRate”:“0.6563203475530601”}
17.Consumer拉取速率
kafka.network:type=RequestMetrics,name=RequestsPerSec,request=FetchConsumer

{“count”:“125796”,“fifteenMinuteRate”:“1.14193044007404E-33”,“fiveMinuteRate”:“7.699516480260211E-100”,“meanRate”:“0.7623419964866819”,“oneMinuteRate”:“2.964393875E-314”}
18.Follower拉去速率
kafka.network:type=RequestMetrics,name=RequestsPerSec,request=FetchFollower

{“count”:“375108”,“fifteenMinuteRate”:“2.302746562040189”,“fiveMinuteRate”:“2.292459728166488”,“meanRate”:“2.2721808581484693”,“oneMinuteRate”:“2.2814260196672973”}
19.Request total time
kafka.network:type=RequestMetrics,name=TotalTimeMs,request=Produce

{“50thPercentile”:“1.0”,“75thPercentile”:“1.0”,“95thPercentile”:“2.0”,“98thPercentile”:“2.0”,“999thPercentile”:“28.0”,“99thPercentile”:“4.0”,“count”:“83384”,“max”:“48.0”,“mean”:“1.2344934279957787”,“min”:“0.0”,“stdDev”:“1.1783192073287214”}
20.Consumer fetch total time
kafka.network:type=RequestMetrics,name=TotalTimeMs,request=FetchConsumer

{“50thPercentile”:“500.0”,“75thPercentile”:“501.0”,“95thPercentile”:“501.0”,“98thPercentile”:“501.0”,“999thPercentile”:“501.971”,“99thPercentile”:“501.0”,“count”:“125796”,“max”:“535.0”,“mean”:“499.83123469744663”,“min”:“0.0”,“stdDev”:“17.138716708632025”}
21.Follower fetch total time
kafka.network:type=RequestMetrics,name=TotalTimeMs,request=FetchFollower

{“50thPercentile”:“500.0”,“75thPercentile”:“500.0”,“95thPercentile”:“501.0”,“98thPercentile”:“501.0”,“999thPercentile”:“507.826”,“99thPercentile”:“501.0”,“count”:“375564”,“max”:“532.0”,“mean”:“437.79763502359117”,“min”:“0.0”,“stdDev”:“148.25999023472986”}
22.Time the follower fetch request waits in the request queue
kafka.network:type=RequestMetrics,name=RequestQueueTimeMs,request=FetchFollower

{“50thPercentile”:“0.0”,“75thPercentile”:“0.0”,“95thPercentile”:“0.0”,“98thPercentile”:“0.0”,“999thPercentile”:“0.0”,“99thPercentile”:“0.0”,“count”:“376206”,“max”:“28.0”,“mean”:“0.0010260336092459982”,“min”:“0.0”,“stdDev”:“0.1282889653905258”}
23.Time the Consumer fetch request waits in the request queue
kafka.network:type=RequestMetrics,name=RequestQueueTimeMs,request=FetchConsumer

{“50thPercentile”:“0.0”,“75thPercentile”:“0.0”,“95thPercentile”:“0.0”,“98thPercentile”:“0.0”,“999thPercentile”:“0.0”,“99thPercentile”:“0.0”,“count”:“125796”,“max”:“24.0”,“mean”:“0.0018124582657636174”,“min”:“0.0”,“stdDev”:“0.18122860552537737”}
24.Time the Produce fetch request waits in the request queue
kafka.network:type=RequestMetrics,name=RequestQueueTimeMs,request=Produce

{“50thPercentile”:“0.0”,“75thPercentile”:“0.0”,“95thPercentile”:“0.0”,“98thPercentile”:“0.0”,“999thPercentile”:“0.0”,“99thPercentile”:“0.0”,“count”:“83704”,“max”:“12.0”,“mean”:“2.6283092803211315E-4”,“min”:“0.0”,“stdDev”:“0.042892540270754634”}
25.Broker I/O工作处理线程空闲率
kafka.network:type=SocketServer,name=NetworkProcessorAvgIdlePercent

{“value”:“1.0015540075894207”}
26.ISR变化速率
kafka.server:type=ReplicaManager,name=IsrShrinksPerSec

{“count”:“0”,“fifteenMinuteRate”:“0.0”,“fiveMinuteRate”:“0.0”,“meanRate”:“0.0”,“oneMinuteRate”:"0

Kafka可以配置使用JMX进行运行状态的监控，既可以通过JDK自带Jconsole来观察结果，也可以通过Java API的方式来.
关于监控指标的描述，可以参考： http://kafka.apache.org/documentation.html#monitoring

开启JMX端口
修改bin/kafka-server-start.sh，添加JMX_PORT参数，添加后样子如下
if [ “x$KAFKA_HEAP_OPTS” = “x” ]; then
export KAFKA_HEAP_OPTS="-Xmx1G -Xms1G"
export JMX_PORT=“9999”
fi

通过Jconsole测试时候可以连接

通过JavaAPI来访问

通过以下方法获取目标值
public class KafkaDataProvider{
protected final Logger LOGGER = LoggerFactory.getLogger(getClass());
private static final String MESSAGE_IN_PER_SEC = “kafka.server:type=BrokerTopicMetrics,name=MessagesInPerSec”;
private static final String BYTES_IN_PER_SEC = “kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec”;
private static final String BYTES_OUT_PER_SEC = “kafka.server:type=BrokerTopicMetrics,name=BytesOutPerSec”;
private static final String PRODUCE_REQUEST_PER_SEC = “kafka.network:type=RequestMetrics,name=RequestsPerSec,request=Produce”;
private static final String CONSUMER_REQUEST_PER_SEC = “kafka.network:type=RequestMetrics,name=RequestsPerSec,request=FetchConsumer”;
private static final String FLOWER_REQUEST_PER_SEC = “kafka.network:type=RequestMetrics,name=RequestsPerSec,request=FetchFollower”;
private static final String ACTIVE_CONTROLLER_COUNT = “kafka.controller:type=KafkaController,name=ActiveControllerCount”;
private static final String PART_COUNT = “kafka.server:type=ReplicaManager,name=PartitionCount”;
public String extractMonitorData() {
//TODO 通过调用API获得IP以及参数
KafkaRoleInfo monitorDataPoint = new KafkaRoleInfo();
String jmxURL = “service:jmx:rmi:///jndi/rmi://192.168.40.242:9999/jmxrmi”;
try {
MBeanServerConnection jmxConnection = MetricDataUtils.getMBeanServerConnection(jmxURL);
ObjectName messageCountObj = new ObjectName(MESSAGE_IN_PER_SEC);
ObjectName bytesInPerSecObj = new ObjectName(BYTES_IN_PER_SEC);
ObjectName bytesOutPerSecObj = new ObjectName(BYTES_OUT_PER_SEC);
ObjectName produceRequestsPerSecObj = new ObjectName(PRODUCE_REQUEST_PER_SEC);
ObjectName consumerRequestsPerSecObj = new ObjectName(CONSUMER_REQUEST_PER_SEC);
ObjectName flowerRequestsPerSecObj = new ObjectName(FLOWER_REQUEST_PER_SEC);
ObjectName activeControllerCountObj = new ObjectName(ACTIVE_CONTROLLER_COUNT);
ObjectName partCountObj = new ObjectName(PART_COUNT);
Long messagesInPerSec = (Long) jmxConnection.getAttribute(messageCountObj, “Count”);
Long bytesInPerSec = (Long) jmxConnection.getAttribute(bytesInPerSecObj, “Count”);
Long bytesOutPerSec = (Long) jmxConnection.getAttribute(bytesOutPerSecObj, “Count”);
Long produceRequestCountPerSec = (Long) jmxConnection.getAttribute(produceRequestsPerSecObj, “Count”);
Long consumerRequestCountPerSec = (Long) jmxConnection.getAttribute(consumerRequestsPerSecObj, “Count”);
Long flowerRequestCountPerSec = (Long) jmxConnection.getAttribute(flowerRequestsPerSecObj, “Count”);
Integer activeControllerCount = (Integer) jmxConnection.getAttribute(activeControllerCountObj, “Value”);
Integer partCount = (Integer) jmxConnection.getAttribute(partCountObj, “Value”);
monitorDataPoint.setMessagesInPerSec(messagesInPerSec);
monitorDataPoint.setBytesInPerSec(bytesInPerSec);
monitorDataPoint.setBytesOutPerSec(bytesOutPerSec);
monitorDataPoint.setProduceRequestCountPerSec(produceRequestCountPerSec);
monitorDataPoint.setConsumerRequestCountPerSec(consumerRequestCountPerSec);
monitorDataPoint.setFlowerRequestCountPerSec(flowerRequestCountPerSec);
monitorDataPoint.setActiveControllerCount(activeControllerCount);
monitorDataPoint.setPartCount(partCount);
} catch (IOException e) {
e.printStackTrace();
} catch (MalformedObjectNameException e) {
e.printStackTrace();
} catch (AttributeNotFoundException e) {
e.printStackTrace();
} catch (MBeanException e) {
e.printStackTrace();
} catch (ReflectionException e) {
e.printStackTrace();
} catch (InstanceNotFoundException e) {
e.printStackTrace();
}
return monitorDataPoint.toString();
}
public static void main(String[] args) {
System.out.println(new KafkaDataProvider().extractMonitorData());
}
/**
* 获得MBeanServer 的连接
*
* @param jmxUrl
* @return
* @throws IOException
*/
public MBeanServerConnection getMBeanServerConnection(String jmxUrl) throws IOException {
JMXServiceURL url = new JMXServiceURL(jmxUrl);
JMXConnector jmxc = JMXConnectorFactory.connect(url, null);
MBeanServerConnection mbsc = jmxc.getMBeanServerConnection();
return mbsc;
}
}

其他工具
除了自己编写定制化的监控程序外

kafka-web-console
https://github.com/claudemamo/kafka-web-console
部署sbt：
http://www.scala-sbt.org/0.13/tutorial/Manual-Installation.html
http://www.scala-sbt.org/release/tutorial/zh-cn/Installing-sbt-on-Linux.html

KafkaOffsetMonitor
https://github.com/quantifind/KafkaOffsetMonitor/releases/tag/v0.2.0
java -cp KafkaOffsetMonitor-assembly-0.2.0.jar com.quantifind.kafka.offsetapp.OffsetGetterWeb --zk localhost:12181 --port 8080 --refresh 5.minutes --retain 1.day