Apache Pulsar消息清理流程源码分析

---

提要

Apache Bookkeeper github地址：https://github.com/apache/bookkeeper
本源码分析基于bookkeeper 4.14版本
Apache Pulsar github地址：https://github.com/apache/pulsar
本源码分析基于pulsar 2.8.2版本

术语说明

0.bk: Bookkeeper中bookie节点的简称
1.ledger/fragment/entry: ledge是Bookkeeper中分布式日志分段，由多个fragment组成。fragment由多个entry组成。

一、背景

在Pulsar中，当生产者向topic分区生产消息时，由Broker中的ManagerLedger组件负责将消息写入该分区最新的ledger中，topic分区的元数据存储在Pulsar的元数据存储zk中。写ledger时bk client会根据所配置的副本策略(E-qw-qa)写入ensemble对应的bk中，其中ledger的元数据存储在bookkeeper的元数据存储zk中。
Pulsar的topic分区元数据存储在zk的managed-ledgers/public/default/persistent/${topicName}-partition-${partitionIndex}目录节点，元数据详情如下：

Bookkeeper的Ledger元数据存储在zk的ledgers/层级目录下，元数据详情如下：

bk提供接口获取ledger元数据信息，接口类如下：

@LimitedPrivate
@Unstable
public interface LedgerMetadata {
   
    /**
     * Returns the id of this ledger.
     *
     * @return the id of this ledger.
     */
    long getLedgerId();

    /**
     * Returns the ensemble size of this ledger.
     *
     * @return the ensemble size of this ledger.
     */
    int getEnsembleSize();

    /**
     * Returns the write quorum size of this ledger.
     *
     * @return the write quorum size of this ledger.
     */
    int getWriteQuorumSize();

    /**
     * Returns the ack quorum size of this ledger.
     *
     * @return the ack quorum size of this ledger.
     */
    int getAckQuorumSize();

    /**
     * Returns the last entry id of this ledger.
     *
     * <p>If this ledger is not sealed {@link #isClosed()}, it returns {@code -1L}.
     *
     * @return the last entry id of this ledger if it is sealed, otherwise -1.
     */
    long getLastEntryId();

    /**
     * Returns the length of this ledger.
     *
     * <p>If this ledger is not sealed {@link #isClosed()}, it returns {@code 0}.
     *
     * @return the length of this ledger if it is sealed, otherwise 0.
     */
    long getLength();

    /**
     * Whether the metadata contains the password and digest type for the ledger.
     * Ledgers created with version 4.1.0 clients or older do not have this information.
     *
     * @return true if the metadata contains the password and digest type, false otherwise.
     */
    boolean hasPassword();

    /**
     * Get the password for the ledger.
     * For ledgers created with version 4.1.0 or older, an empty byte array is returned.
     *
     * @return the password for the ledger.
     */
    byte[] getPassword();

    /**
     * Returns the digest type used by this ledger.
     * May return null if the ledger was created with version 4.1.0 or below.
     *
     * @return the digest type used by this ledger.
     */
    DigestType getDigestType();

    /**
     * Returns the creation timestamp of this ledger.
     *
     * @return the creation timestamp of this ledger.
     */
    long getCtime();

    /**
     * Returns whether the ledger is sealed or not.
     *
     * @return true if the ledger is sealed, otherwise false.
     */
    boolean isClosed();

    /**
     * Returns the custom metadata stored with the ledgers.
     *
     * @return the custom metadata stored with the ledgers.
     */
    Map<String, byte[]> getCustomMetadata();

    /**
     * Returns the ensemble at the given {@code entryId}.
     *
     * @param entryId the entry id to retrieve its ensemble information
     * @return the ensemble which contains the given {@code entryId}.
     */
    List<BookieId> getEnsembleAt(long entryId);

    /**
     * Returns all the ensembles of this ledger.
     *
     * @return all the ensembles of this ledger.
     */
    NavigableMap<Long, ? extends List<BookieId>> getAllEnsembles()