配置openfalcon监控kafka lag堆积情况

1. 尝试通过JMX_PORT查看kafka当前集群状态

# 若JMX_PORT已被持续启用,执行kafka-consumer-groups.sh会报错
kafka-consumer-groups.sh --describe --bootstrap-server localhost:9092 --all-groups 

Error: Exception thrown by the agent : java.rmi.server.ExportException: Port already in use: 9092; nested exception is:
    java.net.BindException: Address in use (Bind failed)

解决方法：

cp /opt/kafka/bin/kafka-run-class.sh /opt/kafka/bin/kafka-run-class.sh.bk
sed -i '47i # need to check if called to start server or client\n# in order to correctly decide about JMX_PORT\nISKAFKASERVER="false"\nif [[ "$*" =~ "kafka.Kafka" ]]; then\nISKAFKASERVER="true"\nfi\n' /opt/kafka/bin/kafka-run-class.sh
sed -i 's#JMX_PORT ]; then#JMX_PORT ] \&\& [ -z "$ISKAFKASERVER" ]; then#g' /opt/kafka/bin/kafka-run-class.sh

参考：
https://github.com/apache/kafka/pull/1983/commits/2c5d40e946bcc149b1a9b2c01eced4ae47a734c5

2. 再次执行，查看结果

kafka-consumer-groups.sh --describe --bootstrap-server localhost:9092 --all-groups # 查看所有GROUP
# kafka-consumer-groups.sh --describe --bootstrap-server localhost:9092 --group monit # 查看指定GROUP
# kafka-consumer-groups.sh --describe --bootstrap-server localhost:9092 --group log # 查看指定GROUP

执行结果，仅供参考：

GROUP|TOPIC|PARTITION|CURRENT-OFFSET|LOG-END-OFFSET|LAG|CONSUMER-ID|HOST|CLIENT-ID|
-|-|-|-|-|-|-|-|-|
log | log-topic | 2 | 16558393 | 16558600 | 207 | logstash-3-beedb2fa-75a1-4bd2-844c-f1b8ba02202f | /10.51.11.102 | logstash-3 |
log | log-topic | 3 | 16558425 | 16558640 | 215 | logstash-1-fc2f47db-8e38-4b33-bf20-2cb04bfbb267 | /10.51.11.102 | logstash-1 |
log | log-topic | 1 | 16554016 | 16554213 | 197 | logstash-3-44b15f1a-c9fb-4638-bd11-664285646a10 | /10.51.11.2   | logstash-3 |
log | log-topic | 0 | 16556708 | 16556915 | 207 | logstash-1-43493a78-cb51-41eb-966f-4f81f7698104 | /10.51.11.2   | logstash-1 |
monit | monit-topic | 2 | 16554367 | 16554552 | 185 | logstash-2-555cc383-7f5b-4e55-8a57-e5b2a253a5a9 | /10.51.11.102 | logstash-2 |
monit | monit-topic | 3 | 16554336 | 16554531 | 195 | logstash-2-b0618d34-d26b-41d5-be01-3f91f1f6e921 | /10.51.11.2   | logstash-2 |
monit | monit-topic | 1 | 16551651 | 16551882 | 231 | logstash-0-940b2e9d-a27f-46db-a9ee-655d940f34ae | /10.51.11.2   | logstash-0 |
monit | monit-topic | 0 | 16786922 | 16787127 | 205 | logstash-0-8b7cc6a0-f3b3-4de6-a6bb-51556b4d4ccb | /10.51.11.102 | logstash-0 |

3. 从执行结果中取出需要的值(lag)

# 在kafka 容器中执行：
lags=$(kafka-consumer-groups.sh --describe --bootstrap-server localhost:9092 --all-groups | grep -v LAG| awk '{print $6}'); echo $lags;echo $lags|xargs -n1 | awk '{sum+=$1} END{print sum}'
41 48 45 35 34 22 44 54 0 0 0 0 0 0 0 0 # 各个当前lag的值
323 # 求和
exit

# 由于需要在kafka容器外执行，那么改写一下：
docker exec kafka bash -c "lags=\$(kafka-consumer-groups.sh --describe --bootstrap-server localhost:9092 --all-groups | grep -v LAG| awk '{print \$6}'); echo \$lags;echo \$lags|xargs -n1 | awk '{sum+=\$1} END{print sum}'"
173 150 152 158 178 169 177 163 0 1 0 0 1 1 0 2
1325

4. crontab定时取值推送给openfalcon

#!/bin/bash
# crontab :  * * * * * /opt/ops/kafka_lag_monit_openfalcon.sh
# 取出想监控的值，尝试push给openfalcon的agent
# 参考：http://book.open-falcon.com/zh_0_2/distributed_install/agent.html

value=$(docker exec kafka bash -c "kafka-consumer-groups.sh --describe --bootstrap-server localhost:9092 --all-groups | grep -v LAG| awk '{print \$6}' | awk '{sum+=\$1} END{print sum}'")
curl -sX POST http://open-falcon-ip:1988/v1/push -d "[{\"counterType\":\"GAUGE\",\"endpoint\":\"${HOSTNAME}\",\"metric\":\"kafka.lag.sum\",\"step\":60,\"tags\":\"kafka\",\"timestamp\":$(date +%s),\"value\":${value}}]" > /dev/null

参考：

http://trumandu.github.io/2019/04/13/如何监控kafka消费Lag情况/