不愧是Alibaba技术官，Kafka的精髓全写这本“限量笔记，附Java面经

• 分区数的上限
• 考量因素

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-4AQ8ZRRs-1634827978389)(//upload-images.jianshu.io/upload_images/19471645-7d1754beb1f7ca5f?imageMogr2/auto-orient/strip|imageView2/2/w/1200/format/webp)]

五、日志存储

1.文件目录布局

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-ySYlDobF-1634827978391)(//upload-images.jianshu.io/upload_images/19471645-cdd97fba80b44aad?imageMogr2/auto-orient/strip|imageView2/2/w/1200/format/webp)]

2.日志格式的演变

• v0版本
• v1版本
• 消息压缩
• 变长字段
• v2版本

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-lCFml9yw-1634827978392)(//upload-images.jianshu.io/upload_images/19471645-f7d9f5958a15a22f?imageMogr2/auto-orient/strip|imageView2/2/w/1200/format/webp)]

3.日志索引

• 偏移量索引
• 时间戳索引

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-137McOWt-1634827978394)(//upload-images.jianshu.io/upload_images/19471645-7d7ee83783569939?imageMogr2/auto-orient/strip|imageView2/2/w/1200/format/webp)]

4.日志清理

• 日志删除
• 日志压缩

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-Li3GUYrC-1634827978396)(//upload-images.jianshu.io/upload_images/19471645-e8a26c50cfacdeed.png?imageMogr2/auto-orient/strip|imageView2/2/w/1200/format/webp)]

5.磁盘存储

• 页缓存
• 磁盘I/O流程
• 零拷贝

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-RQDgY9pX-1634827978397)(//upload-images.jianshu.io/upload_images/19471645-9f60cb1c5fba325f?imageMogr2/auto-orient/strip|imageView2/2/w/1200/format/webp)]

六、深入服务端

1.协议设计

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-OE4a93EB-1634827978398)(//upload-images.jianshu.io/upload_images/19471645-b3706e3bfe2de74a.png?imageMogr2/auto-orient/strip|imageView2/2/w/1200/format/webp)]

2.时间轮

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-be74Ia2P-1634827978399)(//upload-images.jianshu.io/upload_images/19471645-4c79d6d0306a1f75?imageMogr2/auto-orient/strip|imageView2/2/w/1200/format/webp)]

3.延时操作

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-LqBLY93X-1634827978400)(//upload-images.jianshu.io/upload_images/19471645-c3fd6a670d5ec620?imageMogr2/auto-orient/strip|imageView2/2/w/1200/format/webp)]

4.控制器

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-WQTqB5AA-1634827978400)(//upload-images.jianshu.io/upload_images/19471645-094079dbd2484bab.png?imageMogr2/auto-orient/strip|imageView2/2/w/1200/format/webp)]

5.参数解密

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-znBtLPNB-1634827978401)(//upload-images.jianshu.io/upload_images/19471645-c16a9fdbf9963f17?imageMogr2/auto-orient/strip|imageView2/2/w/1200/format/webp)]

七、深入客户端

1.分区分配策略

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-lWatpjnb-1634827978402)(//upload-images.jianshu.io/upload_images/19471645-1211d3a5c380caad?imageMogr2/auto-orient/strip|imageView2/2/w/1200/format/webp)]

2.消费者协调器和组协调器

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-X2vnlWse-1634827978402)(//upload-images.jianshu.io/upload_images/19471645-31cbd96ea10a49c8?imageMogr2/auto-orient/strip|imageView2/2/w/1200/format/webp)]

3._consumer_offsets剖析

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-2gkYcom0-1634827978403)(//upload-images.jianshu.io/upload_images/19471645-d04e04ebf9cec3d0?imageMogr2/auto-orient/strip|imageView2/2/w/1200/format/webp)]

4.事务

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-Y7VOKF1y-1634827978404)(//upload-images.jianshu.io/upload_images/19471645-c6782653edc1556a?imageMogr2/auto-orient/strip|imageView2/2/w/1200/format/webp)]

八、可靠性探究

1.副本剖析

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-XeoQEp4F-1634827978404)(//upload-images.jianshu.io/upload_images/19471645-35adabaa0933b043?imageMogr2/auto-orient/strip|imageView2/2/w/1200/format/webp)]

2.日志同步机制

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-CmpgXqJ7-1634827978405)(//upload-images.jianshu.io/upload_images/19471645-9f999c32f9dd76ce?imageMogr2/auto-orient/strip|imageView2/2/w/1200/format/webp)]

3.可靠性分析

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-kLf55Cwf-1634827978406)(//upload-images.jianshu.io/upload_images/19471645-0d2bcec81905b35d?imageMogr2/auto-orient/strip|imageView2/2/w/1200/format/webp)]

九、Kafka应用

1.命令行工具

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-nvojz3y7-1634827978406)(//upload-images.jianshu.io/upload_images/19471645-03ff9ea1c615fa49?imageMogr2/auto-orient/strip|imageView2/2/w/1200/format/webp)]

2.Kafka Connect

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-n1VjJp8l-1634827978407)(//upload-images.jianshu.io/upload_images/19471645-f08dafc8a0d62cb9?imageMogr2/auto-orient/strip|imageView2/2/w/1200/format/webp)]

3.Kafka Mirror Maker

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-UF5D9jPA-1634827978408)(//upload-images.jianshu.io/upload_images/19471645-add4888231c6e4e8?imageMogr2/auto-orient/strip|imageView2/2/w/1200/format/webp)]

4.Kafka Streams

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-KBRY2aa0-1634827978408)(//upload-images.jianshu.io/upload_images/19471645-c416fb3732501cab?imageMogr2/auto-orient/strip|imageView2/2/w/1200/format/webp)]

十、Kafka监控

1.监控数据的来源

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-gdEkDAMv-1634827978409)(//upload-images.jianshu.io/upload_images/19471645-10f451fd4c6116d9?imageMogr2/auto-orient/strip|imageView2/2/w/1200/format/webp)]

2.消费滞后

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-6hYdsuhS-1634827978409)(//upload-images.jianshu.io/upload_images/19471645-b6bc2f98a50e393a?imageMogr2/auto-orient/strip|imageView2/2/w/1200/format/webp)]

3.同步失效分区

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-hfl2tpJy-1634827978410)(//upload-images.jianshu.io/upload_images/19471645-25bb2fed7d89c3c4?imageMogr2/auto-orient/strip|imageView2/2/w/1200/format/webp)]

4.监控指标说明

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-HIJTg8tO-1634827978410)(//upload-images.jianshu.io/upload_images/19471645-bde3d0887956765a?imageMogr2/auto-orient/strip|imageView2/2/w/1200/format/webp)]

5.监控模块

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-C1y3fBGz-1634827978411)(//upload-images.jianshu.io/upload_images/19471645-134ecb044f1c7a20?imageMogr2/auto-orient/strip|imageView2/2/w/1200/format/webp)]

十一、高级应用

1.过期时间(TTL)

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-dgIAPqRQ-1634827978411)(//upload-images.jianshu.io/upload_images/19471645-c8247ac0d2e8757f?imageMogr2/auto-orient/strip|imageView2/2/w/1200/format/webp)]

2.延时队列

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-1f7EctSX-1634827978412)(//upload-images.jianshu.io/upload_images/19471645-0d5bf5ad7744b105?imageMogr2/auto-orient/strip|imageView2/2/w/1200/format/webp)]

3.死信队列和重试队列

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-Y7nldSWv-1634827978412)(//upload-images.jianshu.io/upload_images/19471645-cb90da0ec9c831af?imageMogr2/auto-orient/strip|imageView2/2/w/1200/format/webp)]

4.消息路由

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-V8GgRXeD-1634827978413)(//upload-images.jianshu.io/upload_images/19471645-4aa0b24282d9a6c5?imageMogr2/auto-orient/strip|imageView2/2/w/1200/format/webp)]

5.消息轨迹

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-LEneBjrL-1634827978413)(//upload-images.jianshu.io/upload_images/19471645-61d914850f9be9bc?imageMogr2/auto-orient/strip|imageView2/2/w/1200/format/webp)]

6.消息审计

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-WQJhkIvL-1634827978414)(//upload-images.jianshu.io/upload_images/19471645-99525ea4144c42f6?imageMogr2/auto-orient/strip|imageView2/2/w/1200/format/webp)]

7.消息代理

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-SOmvNQDA-1634827978414)(//upload-images.jianshu.io/upload_images/19471645-5d9cb30493c9831b?imageMogr2/auto-orient/strip|imageView2/2/w/1200/format/webp)]

8.消息中间件选型

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-K9M6TFOT-1634827978415)(//upload-images.jianshu.io/upload_images/19471645-d0fe2aab69719da7?imageMogr2/auto-orient/strip|imageView2/2/w/1200/format/webp)]

十二、Kafka与Spark的集成

1.Spark的安装及简单应用

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-Nr20NX1M-1634827978416)(//upload-images.jianshu.io/upload_images/19471645-d279440bf7e98df5?imageMogr2/auto-orient/strip|imageView2/2/w/1200/format/webp)]

2.Spark编程模型

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-I4qKhNHJ-1634827978416)(//upload-images.jianshu.io/upload_images/19471645-079042317bdb8704?imageMogr2/auto-orient/strip|imageView2/2/w/1200/format/webp)]

3.Spark的运行结构

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-b3EViuVY-1634827978417)(//upload-images.jianshu.io/upload_images/19471645-df096c73426e427d.png?imageMogr2/auto-orient/strip|imageView2/2/w/1200/format/webp)]

4.Spark Streaming简介

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-1mfS0YHR-1634827978418)(//upload-images.jianshu.io/upload_images/19471645-5ad264e01076e9c6?imageMogr2/auto-orient/strip|imageView2/2/w/1200/format/webp)]

5.Kafka与Spark Streaming的整合

NHJ-1634827978416)]

3.Spark的运行结构

[外链图片转存中…(img-b3EViuVY-1634827978417)]

4.Spark Streaming简介

[外链图片转存中…(img-1mfS0YHR-1634827978418)]

5.Kafka与Spark Streaming的整合