KafkaOffsetMonitor：监控消费者和延迟的队列

参考博客：

https://www.orchome.com/54

安装步骤：

（1）下载软件

KafkaOffsetMonitor-assembly-0.3.0-SNAPSHOT.jar

（2）解压打开：KafkaOffsetMonitor-assembly-0.3.0-SNAPSHOT.jar\offsetapp下面的index.html文件，替换成下面的内容，

不然google的静态文件加载不进来.

<title>Kafka Consumer Offset Monitor</title>

<!-- Bootstrap core CSS -->
<link href="./scripts/bootstrap.min.css" rel="stylesheet">
<link href="./style.css" rel="stylesheet">
<link href="./css/cluster-viz.css" rel="stylesheet">

<script src="./scripts/angular.min.js"></script>
<script src="./scripts/angular-route.js"></script>
<script src="./scripts/angular-resource.js"></script>

<!-- HTML5 shim and Respond.js IE8 support of HTML5 elements and media queries -->
<!--[if lt IE 9]>
  <script src="https://oss.maxcdn.com/libs/html5shiv/3.7.0/html5shiv.js"></script>
  <script src="https://oss.maxcdn.com/libs/respond.js/1.3.0/respond.min.js"></script>
<![endif]-->

Toggle navigation KafkaMonitor

          <!-- Collect the nav links, forms, and other content for toggling -->
          <div class="collapse navbar-collapse" id="bs-example-navbar-collapse-1">
              <ul class="nav navbar-nav">
                  <li><a href="#">Consumer Groups</a></li>
                  <li><a href="/#/topics">Topic List</a></li>
                  <li class="dropdown">
                      <a  href="javascript:void(0)"  class="dropdown-toggle" data-toggle="dropdown">Visualizations <b class="caret"></b></a>
                      <ul class="dropdown-menu">
                          <li><a href="/#/activetopicsviz">Active Topic Consumers</a></li>
                          <li><a href="/#/clusterviz">Cluster Overview</a></li>
                      </ul>
                  </li>
              </ul>
          </div><!-- /.navbar-collapse -->
      </div><!-- /.container-fluid -->
  </nav>
<div class="container">

  <div ng-view></div>

</div><!-- /.container -->


<!-- Bootstrap core JavaScript
================================================== -->
<!-- Placed at the end of the document so the pages load faster -->
<script src="./scripts/jquery-1.10.2.min.js"></script>
<script src="./scripts/moment.min.js"></script>
<script src="./scripts/bootstrap.min.js"></script>
<script src="./scripts/highstock.js"></script>
<script src="./scripts/exporting.js"></script>
<script src="./scripts/lodash.min.js"></script>
<script src="./scripts/d3.v3.min.js"></script>
<script src="./scripts/directives.js"></script>
<script src="./scripts/controllers.js"></script>
<script src="./scripts/app.js"></script>
<script src="./scripts/cluster-viz.js"></script>

（3）将源index.html需要的静态文件下载下来，并放置到KafkaOffsetMonitor-assembly-0.3.0-SNAPSHOT.jar\offsetapp\scripts 下面：

（4）部署jar包，并启动：

java -Xms512M -Xmx512M -Xss1024K -XX:PermSize=256m -XX:MaxPermSize=512m -cp KafkaOffsetMonitor-assembly-0.3.0-SNAPSHOT.jar com.quantifind.kafka.offsetapp.OffsetGetterWeb --offsetStorage kafka --zk 172.21.50.162:2181 --port 8088 --refresh 10.seconds --retain 1.days >offsetStorage.log &

（5）查看页面：

Kafka 0号分区一共有24条记录，已经消费了24条，下一条待消费记录的offset是24！！！

命令行查看一下：

[bdp_client@BJLFRZ-Client-50-162 software]$ /home/bdp_client/zmy/software/kafka/bin/kafka-consumer-groups.sh --new-consumer --bootstrap-server 172.21.50.162:9092 --describe --group topic-review-consumer_group_1
GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG OWNER
topic-review-consumer_group_1 topic-review-1 0 24 24 0 consumer_001_/172.21.50.10
topic-review-consumer_group_1 topic-review-1 1 12 12 0 consumer_001_/172.21.50.10
topic-review-consumer_group_1 topic-review-1 2 9 9 0 consumer_001_/172.21.50.10

查看一下数据文件：

[bdp_client@BJLFRZ-Client-50-162 topic-review-1-2]$ /home/bdp_client/zmy/software/kafka/bin/kafka-run-class.sh kafka.tools.DumpLogSegments --files 00000000000000000000.log --print-data-log
Dumping 00000000000000000000.log
Starting offset: 0
offset: 0 position: 0 CreateTime: 1576818856261 isvalid: true payloadsize: 5 magic: 1 compresscodec: NoCompressionCodec crc: 2777638059 payload: �K.
offset: 1 position: 39 CreateTime: 1576823575277 isvalid: true payloadsize: 5 magic: 1 compresscodec: NoCompressionCodec crc: 1878898881 payload: �K.
offset: 2 position: 78 CreateTime: 1576823575279 isvalid: true payloadsize: 5 magic: 1 compresscodec: NoCompressionCodec crc: 3164937966 payload: �K.
offset: 3 position: 117 CreateTime: 1576823575280 isvalid: true payloadsize: 5 magic: 1 compresscodec: NoCompressionCodec crc: 2509809461 payload: �K.
offset: 4 position: 156 CreateTime: 1576823632113 isvalid: true payloadsize: 5 magic: 1 compresscodec: NoCompressionCodec crc: 1432097505 payload: �K.
offset: 5 position: 195 CreateTime: 1576837946708 isvalid: true payloadsize: 5 magic: 1 compresscodec: NoCompressionCodec crc: 984602471 payload: �K.
offset: 6 position: 234 CreateTime: 1576837946708 isvalid: true payloadsize: 5 magic: 1 compresscodec: NoCompressionCodec crc: 293766308 payload: �K.
offset: 7 position: 273 CreateTime: 1576844955259 isvalid: true payloadsize: 5 magic: 1 compresscodec: NoCompressionCodec crc: 2860390913 payload: �K.
offset: 8 position: 312 CreateTime: 1576844955262 isvalid: true payloadsize: 5 magic: 1 compresscodec: NoCompressionCodec crc: 99624879 payload: �K.
[bdp_client@BJLFRZ-Client-50-163 topic-review-1-0]$ /home/bdp_client/zmy/software/kafka/bin/kafka-run-class.sh kafka.tools.DumpLogSegments --files 00000000000000000000.log --print-data-log

Dumping 00000000000000000000.log
Starting offset: 0
offset: 0 position: 0 CreateTime: 1576812789428 isvalid: true payloadsize: 5 magic: 1 compresscodec: NoCompressionCodec crc: 3209326490 keysize: 4 key: �N. payload: �K.
offset: 1 position: 43 CreateTime: 1576812789428 isvalid: true payloadsize: 5 magic: 1 compresscodec: NoCompressionCodec crc: 2790350555 keysize: 4 key: �N. payload: �K.
offset: 2 position: 86 CreateTime: 1576812789429 isvalid: true payloadsize: 5 magic: 1 compresscodec: NoCompressionCodec crc: 1647229945 keysize: 4 key: �N. payload: �K.
offset: 3 position: 129 CreateTime: 1576812789429 isvalid: true payloadsize: 5 magic: 1 compresscodec: NoCompressionCodec crc: 2067106488 keysize: 4 key: �N. payload: �K.
offset: 4 position: 172 CreateTime: 1576812789429 isvalid: true payloadsize: 5 magic: 1 compresscodec: NoCompressionCodec crc: 880024703 keysize: 4 key: �N. payload: �K.
offset: 5 position: 215 CreateTime: 1576818766270 isvalid: true payloadsize: 5 magic: 1 compresscodec: NoCompressionCodec crc: 3425153245 keysize: 4 key: �N. payload: �K.
offset: 6 position: 258 CreateTime: 1576818766270 isvalid: true payloadsize: 5 magic: 1 compresscodec: NoCompressionCodec crc: 3577512348 keysize: 4 key: �N. payload: �K.
offset: 7 position: 301 CreateTime: 1576818766270 isvalid: true payloadsize: 5 magic: 1 compresscodec: NoCompressionCodec crc: 4262580831 keysize: 4 key: �N. payload: �K.
offset: 8 position: 344 CreateTime: 1576818766271 isvalid: true payloadsize: 5 magic: 1 compresscodec: NoCompressionCodec crc: 140006911 keysize: 4 key: �N. payload: �K.
offset: 9 position: 387 CreateTime: 1576818766271 isvalid: true payloadsize: 5 magic: 1 compresscodec: NoCompressionCodec crc: 1192870712 keysize: 4 key: �N. payload: �K.
offset: 10 position: 430 CreateTime: 1576818856261 isvalid: true payloadsize: 5 magic: 1 compresscodec: NoCompressionCodec crc: 3163837930 payload: �K.
offset: 11 position: 469 CreateTime: 1576818856261 isvalid: true payloadsize: 5 magic: 1 compresscodec: NoCompressionCodec crc: 3939431020 payload: �K.
offset: 12 position: 508 CreateTime: 1576822333844 isvalid: true payloadsize: 5 magic: 1 compresscodec: NoCompressionCodec crc: 3379709634 payload: �K.
offset: 13 position: 547 CreateTime: 1576822333844 isvalid: true payloadsize: 5 magic: 1 compresscodec: NoCompressionCodec crc: 3496572803 payload: �K.
offset: 14 position: 586 CreateTime: 1576822333845 isvalid: true payloadsize: 5 magic: 1 compresscodec: NoCompressionCodec crc: 2136011895 payload: �K.
offset: 15 position: 625 CreateTime: 1576822333845 isvalid: true payloadsize: 5 magic: 1 compresscodec: NoCompressionCodec crc: 806445744 payload: �K.
offset: 16 position: 664 CreateTime: 1576823575280 isvalid: true payloadsize: 5 magic: 1 compresscodec: NoCompressionCodec crc: 3671667186 payload: �K.
offset: 17 position: 703 CreateTime: 1576823632112 isvalid: true payloadsize: 5 magic: 1 compresscodec: NoCompressionCodec crc: 4200960277 payload: �K.
offset: 18 position: 742 CreateTime: 1576823632113 isvalid: true payloadsize: 5 magic: 1 compresscodec: NoCompressionCodec crc: 1279730592 payload: �K.
offset: 19 position: 781 CreateTime: 1576823632113 isvalid: true payloadsize: 5 magic: 1 compresscodec: NoCompressionCodec crc: 50771303 payload: �K.
offset: 20 position: 820 CreateTime: 1576824838251 isvalid: true payloadsize: 5 magic: 1 compresscodec: NoCompressionCodec crc: 4126809583 payload: �K.
offset: 21 position: 859 CreateTime: 1576837946707 isvalid: true payloadsize: 5 magic: 1 compresscodec: NoCompressionCodec crc: 1182354662 payload: �K.
offset: 22 position: 898 CreateTime: 1576844955260 isvalid: true payloadsize: 5 magic: 1 compresscodec: NoCompressionCodec crc: 3601367424 payload: �K.
offset: 23 position: 937 CreateTime: 1576844955262 isvalid: true payloadsize: 5 magic: 1 compresscodec: NoCompressionCodec crc: 485168878 payload: �K.
[bdp_client@BJLFRZ-Client-50-164 topic-review-1-1]$ /home/bdp_client/zmy/software/kafka/bin/kafka-run-class.sh kafka.tools.DumpLogSegments --files 00000000000000000000.log --print-data-log

Dumping 00000000000000000000.log
Starting offset: 0
offset: 0 position: 0 CreateTime: 1576818856260 isvalid: true payloadsize: 5 magic: 1 compresscodec: NoCompressionCodec crc: 330159646 payload: �K.
offset: 1 position: 39 CreateTime: 1576818856261 isvalid: true payloadsize: 5 magic: 1 compresscodec: NoCompressionCodec crc: 2545485353 payload: �K.
offset: 2 position: 78 CreateTime: 1576822333845 isvalid: true payloadsize: 5 magic: 1 compresscodec: NoCompressionCodec crc: 1716241718 payload: �K.
offset: 3 position: 117 CreateTime: 1576823575278 isvalid: true payloadsize: 5 magic: 1 compresscodec: NoCompressionCodec crc: 176663643 payload: �K.
offset: 4 position: 156 CreateTime: 1576823632112 isvalid: true payloadsize: 5 magic: 1 compresscodec: NoCompressionCodec crc: 3816726612 payload: �K.
offset: 5 position: 195 CreateTime: 1576824838249 isvalid: true payloadsize: 5 magic: 1 compresscodec: NoCompressionCodec crc: 227534851 payload: �K.
offset: 6 position: 234 CreateTime: 1576824838251 isvalid: true payloadsize: 5 magic: 1 compresscodec: NoCompressionCodec crc: 3738652204 payload: �K.
offset: 7 position: 273 CreateTime: 1576824838251 isvalid: true payloadsize: 5 magic: 1 compresscodec: NoCompressionCodec crc: 3352067949 payload: �K.
offset: 8 position: 312 CreateTime: 1576824838251 isvalid: true payloadsize: 5 magic: 1 compresscodec: NoCompressionCodec crc: 2291001770 payload: �K.
offset: 9 position: 351 CreateTime: 1576837946708 isvalid: true payloadsize: 5 magic: 1 compresscodec: NoCompressionCodec crc: 144291301 payload: �K.
offset: 10 position: 390 CreateTime: 1576837946709 isvalid: true payloadsize: 5 magic: 1 compresscodec: NoCompressionCodec crc: 3671573076 payload: �K.
offset: 11 position: 429 CreateTime: 1576844955262 isvalid: true payloadsize: 5 magic: 1 compresscodec: NoCompressionCodec crc: 1253159272 payload: �K.

从数据文件可以看出，kafka提交的offset下一条待消费记录的offset！