kafka从0.9版本之后引入了配额机制,对于每个producer或者consumer,可以对他们produce或者consum的速度上限作出限制.这边找到一篇不错的文档,可以解释这一配额的设计理念与实现方式.
这其实对应着kafka中的两个线程,限制producer的线程,以及限制consumer的线程.
"ThrottledRequestReaper-Fetch" #24 prio=5 os_prio=0 tid=0x00007fadb4ad2800 nid=0x11a61 waiting on condition [0x00007fac97ffe000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000c86003a8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.DelayQueue.poll(DelayQueue.java:259)
at kafka.server.ClientQuotaManager$ThrottledRequestReaper.doWork(ClientQuotaManager.scala:158)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
"ThrottledRequestReaper-Produce" #25 prio=5 os_prio=0 tid=0x00007fadb4ad7000 nid=0x11a62 waiting on condition [0x00007fac97efd000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000c85e33c8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.DelayQueue.poll(DelayQueue.java:259)
at kafka.server.ClientQuotaManager$ThrottledRequestReaper.doWork(ClientQuotaManager.scala:158)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
我认为QuatoManager是很有学习意义的.起码从两个方面,一是对于kafka这样请求回复式的结构,如何尽可能优雅地实现节流. 二是如何能尽可能优雅地监测一个client的流量.
关于Kafka中的commons.metrics我已经写过一篇文章介绍了,有兴趣的同学可以看这个连接kafka性能监控之KafkaMetrics Sensor
接下来我会介绍一下,具体kafka是如何使用delayQueue来实现Quato功能的,在我刚才给的那个链接里提到了,每一次broker发送response时都会记录下当前consume或者produce了多少字节,如果超出配额就会抛出QuatoVioLationException的异常.
try { clientSensors.quotaSensor.record(value) // trigger the callback immediately if quota is not violated