kafka监控

kafka监控

kafka-web-console

https://github.com/claudemamo/kafka-web-console

部署sbt：

http://www.scala-sbt.org/0.13/tutorial/Manual-Installation.html

http://www.scala-sbt.org/release/tutorial/zh-cn/Installing-sbt-on-Linux.html

KafkaOffsetMonitor

https://github.com/quantifind/KafkaOffsetMonitor/releases/tag/v0.2.0

 

java -cp KafkaOffsetMonitor-assembly-0.2.0.jar com.quantifind.kafka.offsetapp.OffsetGetterWeb --zk localhost:12181 --port 8080 --refresh 5.minutes --retain 1.day

kafka 监控之Mx4jLoader

源码未做任何修改，在lib下添加jar包mx4j-tools-3.0.1

启动server之后访问 ip:8082

/**

  * If mx4j-tools is in the classpath call maybeLoad to load the HTTP interface of mx4j.

  *

  * The default port is 8082. To override that provide e.g. -Dmx4jport=8083

  * The default listen address is 0.0.0.0. To override that provide -Dmx4jaddress=127.0.0.1

  * This feature must be enabled with -Dmx4jenable=true

  *

  * This is a Scala port of org.apache.cassandra.utils.Mx4jTool written by Ran Tavory for CASSANDRA-1068

  * */

JMX监控指标参数列表如下:

 

参数	Mbean名称	说明
Message in rate	"kafka.server":name="AllTopicsMessagesInPerSec",type="BrokerTopicMetrics"	所有topic消息(进出)流量
Byte in rate	"kafka.server":name="AllTopicsBytesInPerSec",type="BrokerTopicMetrics"
Request rate	"kafka.network":name="{Produce|Fetch-consumer|Fetch-follower}-RequestsPerSec",type="RequestMetrics"
Byte out rate	"kafka.server":name="AllTopicsBytesOutPerSec",type="BrokerTopicMetrics"
Log flush rate and time	"kafka.log":name="LogFlushRateAndTimeMs",type="LogFlushStats"
# of under replicated partitions (|ISR| < |all replicas|)	"kafka.server":name="UnderReplicatedPartitions",type="ReplicaManager"	0
Is controller active on broker	"kafka.controller":name="ActiveControllerCount",type="KafkaController"	only one broker in the cluster should have 1
Leader election rate	"kafka.controller":name="LeaderElectionRateAndTimeMs",type="ControllerStats"	non-zero when there are broker failures
Unclean leader election rate	"kafka.controller":name="UncleanLeaderElectionsPerSec",type="ControllerStats"	0
Partition counts	"kafka.server":name="PartitionCount",type="ReplicaManager"	mostly even across brokers
Leader replica counts	"kafka.server":name="LeaderCount",type="ReplicaManager"	mostly even across brokers
ISR shrink rate	"kafka.server":name="ISRShrinksPerSec",type="ReplicaManager"	If a broker goes down, ISR for some of the partitions will shrink. When that broker is up again, ISR will be expanded once the replicas are fully caught up. Other than that, the expected value for both ISR shrink rate and expansion rate is 0.
ISR expansion rate	"kafka.server":name="ISRExpandsPerSec",type="ReplicaManager"	See above
Max lag in messages btw follower and leader replicas	"kafka.server":name="([-.\w]+)-MaxLag",type="ReplicaFetcherManager"	副本消息滞后数量
Lag in messages per follower replica	"kafka.server":name="([-.\w]+)-ConsumerLag",type="FetcherLagMetrics"	副本消息滞后数量
Requests waiting in the producer purgatory	"kafka.server":name="PurgatorySize",type="ProducerRequestPurgatory"
Requests waiting in the fetch purgatory	"kafka.server":name="PurgatorySize",type="FetchRequestPurgatory"
Request total time	"kafka.network":name="{Produce|Fetch-Consumer|Fetch-Follower}-TotalTimeMs",type="RequestMetrics"
Time the request waiting in the request queue	"kafka.network":name="{Produce|Fetch-Consumer|Fetch-Follower}-QueueTimeMs",type="RequestMetrics"
Time the request being processed at the leader	"kafka.network":name="{Produce|Fetch-Consumer|Fetch-Follower}-LocalTimeMs",type="RequestMetrics"
Time the request waits for the follower	"kafka.network":name="{Produce|Fetch-Consumer|Fetch-Follower}-RemoteTimeMs",type="RequestMetrics"
Time to send the response	"kafka.network":name="{Produce|Fetch-Consumer|Fetch-Follower}-ResponseSendTimeMs",type="RequestMetrics"
Number of messages the consumer lags behind the producer by	"kafka.consumer":name="([-.\w]+)-MaxLag",type="ConsumerFetcherManager"