Kafka压缩详解

若文章对你有帮助请点个👍👍👍, 表示对我的鼓励, 非常感谢 ❤️❤️❤️

概括

需要理解kafka压缩则需要理解Kafka的存储格式.

Kafka存储格式

RecordBatch

baseOffset: int64
batchLength: int32
partitionLeaderEpoch: int32
magic: int8 (current magic value is 2)
crc: int32
attributes: int16
    bit 0~2:
        0: no compression
        1: gzip
        2: snappy
        3: lz4
        4: zstd
    bit 3: timestampType
    bit 4: isTransactional (0 means not transactional)
    bit 5: isControlBatch (0 means not a control batch)
    bit 6~15: unused
lastOffsetDelta: int32
firstTimestamp: int64
maxTimestamp: int64
producerId: int64
producerEpoch: int16
baseSequence: int32
records: [Record]

Record

length: varint
attributes: int8
    bit 0