kafka线程模型之三 QuotaManager

kafka从0.9版本之后引入了配额机制,对于每个producer或者consumer,可以对他们produce或者consum的速度上限作出限制.这边找到一篇不错的文档,可以解释这一配额的设计理念与实现方式.

kafka配额控制

这其实对应着kafka中的两个线程,限制producer的线程,以及限制consumer的线程.

"ThrottledRequestReaper-Fetch" #24 prio=5 os_prio=0 tid=0x00007fadb4ad2800 nid=0x11a61 waiting on condition [0x00007fac97ffe000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x00000000c86003a8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
	at java.util.concurrent.DelayQueue.poll(DelayQueue.java:259)
	at kafka.server.ClientQuotaManager$ThrottledRequestReaper.doWork(ClientQuotaManager.scala:158)
	at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)

"ThrottledRequestReaper-Produce" #25 prio=5 os_prio=0 tid=0x00007fadb4ad7000 nid=0x11a62 waiting on condition [0x00007fac97efd000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x00000000c85e33c8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
	at java.util.concurrent.DelayQueue.poll(DelayQueue.java:259)
	at kafka.server.ClientQuotaManager$ThrottledRequestReaper.doWork(ClientQuotaManager.scala:158)
	at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)

我认为QuatoManager是很有学习意义的.起码从两个方面,一是对于kafka这样请求回复式的结构,如何尽可能优雅地实现节流. 二是如何能尽可能优雅地监测一个client的流量.

关于Kafka中的commons.metrics我已经写过一篇文章介绍了,有兴趣的同学可以看这个连接kafka性能监控之KafkaMetrics Sensor

接下来我会介绍一下,具体kafka是如何使用delayQueue来实现Quato功能的,在我刚才给的那个链接里提到了,每一次broker发送response时都会记录下当前consume或者produce了多少字节,如果超出配额就会抛出QuatoVioLationException的异常.

try {
  clientSensors.quotaSensor.record(value)
  // trigger the callback immediately if quota is not violated