postgresql源码学习（53）—— vacuum②-lazy vacuum之heap_vacuum_rel函数

一、 table_relation_vacuum函数

1. 函数定义

      前篇最后（https://blog.csdn.net/Hehuyi_In/article/details/128749517），我们提到了table_relation_vacuum函数（tableam.h文件），本篇继续学习。

       如前面所说，手动和autovacuum触发的vacuum操作均会走到该函数，需要对表加4级锁。该函数针对lazy vacuum，因此vacuum full,CLUSTER,ANALYZE操作不会走到它。

static inline void
table_relation_vacuum(Relation rel, struct VacuumParams *params,
                      BufferAccessStrategy bstrategy)
{
    rel->rd_tableam->relation_vacuum(rel, params, bstrategy);
}

       这里遇到一个问题，relation_vacuum实际不是一个函数，源码文件中也没找到它的内容，无法再看到下层函数，因此这里借助gdb跟踪一把。

2. 下层函数追踪

b table_relation_vacuum

       可以看到后面实际是调用了heap_vacuum_rel函数（vacuumlazy.c文件）。兜兜转转半天，终于见到了lazy vacuum函数的庐山真面目。但在学习它之前，还是先来看看其中几个预备知识点，避免一脸懵逼。

二、 准备知识

1. TransactionIdPrecedesOrEquals函数

这个老朋友之前学习过，用来比较哪个事务id更旧，原理参考：https://blog.csdn.net/Hehuyi_In/article/details/102869893

/*
 * TransactionIdPrecedesOrEquals --- is id1 logically <= id2?
 */
bool
TransactionIdPrecedesOrEquals(TransactionId id1, TransactionId id2)
{
    int32       diff;

    if (!TransactionIdIsNormal(id1) || !TransactionIdIsNormal(id2))
        return (id1 <= id2);

    diff = (int32) (id1 - id2);
    return (diff <= 0);
}

2. LVRelState结构体

LV指的是Lazy Vacuum，从这个名字可以猜测是与表状态相关的结构体。

typedef struct LVRelState
{
    /* Target heap relation and its indexes */
    Relation    rel;
    Relation   *indrels;
    int         nindexes;

    /* Wraparound failsafe has been triggered? */
    bool        failsafe_active;
    /* Consider index vacuuming bypass optimization? */
    bool        consider_bypass_optimization;

    /* Doing index vacuuming, index cleanup, rel truncation? */
    bool        do_index_vacuuming;
    bool        do_index_cleanup;
    bool        do_rel_truncate;

    /* Buffer access strategy and parallel state */
    BufferAccessStrategy bstrategy;
    LVParallelState *lps;

    /* Statistics from pg_class when we start out */
    BlockNumber old_rel_pages;  /* previous value of pg_class.relpages */
    double      old_live_tuples;    /* previous value of pg_class.reltuples */
    /* rel's initial relfrozenxid and relminmxid */
    TransactionId relfrozenxid;
    MultiXactId relminmxid;

    /* VACUUM operation's cutoff for pruning */
    TransactionId OldestXmin;
    /* VACUUM operation's cutoff for freezing XIDs and MultiXactIds */
    TransactionId FreezeLimit;
    MultiXactId MultiXactCutoff;

    /* Error reporting state */
    char       *relnamespace;
    char       *relname;
    char       *indname;
    BlockNumber blkno;          /* used only for heap operations */
    OffsetNumber offnum;        /* used only for heap operations */
    VacErrPhase phase;

    /*
     * State managed by lazy_scan_heap() follows
     */
    LVDeadTuples *dead_tuples;  /* items to vacuum from indexes */
    BlockNumber rel_pages;      /* total number of pages */
    BlockNumber scanned_pages;  /* number of pages we examined */
    BlockNumber pinskipped_pages;   /* # of pages skipped due to a pin */
    BlockNumber frozenskipped_pages;    /* # of frozen pages we skipped */
    BlockNumber tupcount_pages; /* pages whose tuples we counted */
    BlockNumber pages_removed;  /* pages remove by truncation */
    BlockNumber lpdead_item_pages;  /* # pages with LP_DEAD items */
    BlockNumber nonempty_pages; /* actually, last nonempty page + 1 */

    /* Statistics output by us, for table */
    double      new_rel_tuples; /* new estimated total # of tuples */
    double      new_live_tuples;    /* new estimated total # of live tuples */
    /* Statistics output by index AMs */
    IndexBulkDeleteResult **indstats;

    /* Instrumentation counters */
    int         num_index_scans;
    int64       tuples_deleted; /* # deleted from table */
    int64       lpdead_items;   /* # deleted from indexes */
    int64       new_dead_tuples;    /* new estimated total # of dead items in
                                     * table */
    int64       num_tuples;     /* total number of nonremovable tuples */
    int64       live_tuples;    /* live tuples (reltuples estimate) */
} LVRelState;

三、 heap_vacuum_rel函数

       根据注释，该函数负责vacuum单个堆表、清理其索引、更新relpages,reltuples的统计信息。进入此函数前，我们已经完成了事务开启以及对应表的4级锁获取。

/*
 *  heap_vacuum_rel() -- perform VACUUM for one heap relation
 *
 *      This routine vacuums a single heap, cleans out its indexes, and
 *      updates its relpages and reltuples statistics.
 *
 *      At entry, we have already established a transaction and opened
 *      and locked the relation.
 */
void
heap_vacuum_rel(Relation rel, VacuumParams *params,
                BufferAccessStrategy bstrategy)
{
    LVRelState *vacrel;
    PGRUsage    ru0;
    TimestampTz starttime = 0;
    WalUsage    walusage_start = pgWalUsage;
    WalUsage    walusage = {0, 0, 0};
    long        secs;
    int         usecs;
    double      read_rate,
                write_rate;
    bool        aggressive;     /* should we scan all unfrozen pages? 是否应该扫描所有未冻结页？ */
    bool        scanned_all_unfrozen;   /* actually scanned all such pages? 是否实际扫描了所有未冻结页？ */
    char      **indnames = NULL;
    TransactionId xidFullScanLimit;
    MultiXactId mxactFullScanLimit;
    BlockNumber new_rel_pages;
    BlockNumber new_rel_allvisible;
    double      new_live_tuples;
    TransactionId new_frozen_xid;
    MultiXactId new_min_multi;
    ErrorContextCallback errcallback;
    PgStat_Counter startreadtime = 0;
    PgStat_Counter startwritetime = 0;
    TransactionId OldestXmin;
    TransactionId FreezeLimit;
    MultiXactId MultiXactCutoff;
…

       首先根据输入的freeze参数，计算并赋值给各类限制值变量（带&的都是），用于下面判断是否采取迫切模式（aggressive）。根据前面的注释，aggressive=true则需要扫描所有未冻结页。冻结相关参考：postgresql_internals-14 学习笔记（三）冻结、rebuild_Hehuyi_In的博客-CSDN博客

 /* 根据输入的freeze参数，计算各类限制值（带&的都是），用于下面判断是否采取迫切（aggressive）清理 */
    vacuum_set_xid_limits(rel,
                          params->freeze_min_age,
                          params->freeze_table_age,
                          params->multixact_freeze_min_age,
                          params->multixact_freeze_table_age,
                          &OldestXmin, &FreezeLimit, &xidFullScanLimit,
                          &MultiXactCutoff, &mxactFullScanLimit);

    /*     
      * 如果表的relfrozenxid <= xidFullScanLimit（表中最新xid- vacuum_freeze_table_age），则触发aggressive scan，multiXid类似；如果设置了DISABLE_PAGE_SKIPPING（禁用跳过页），则也触发aggressive scan。
     */
    aggressive = TransactionIdPrecedesOrEquals(rel->rd_rel->relfrozenxid,
                                               xidFullScanLimit);
    aggressive |= MultiXactIdPrecedesOrEquals(rel->rd_rel->relminmxid,
                                              mxactFullScanLimit);
    if (params->options & VACOPT_DISABLE_PAGE_SKIPPING)
        aggressive = true;

初始化vacrel变量并根据各类option设置其字段初始值，各字段含义参考LVRelState结构体定义。

vacrel = (LVRelState *) palloc0(sizeof(LVRelState));

    /* Set up high level stuff about rel */
    vacrel->rel = rel;
    /* 打开表索引，返回索引名和数量 */
    vac_open_indexes(vacrel->rel, RowExclusiveLock, &vacrel->nindexes,
                     &vacrel->indrels);
    vacrel->failsafe_active = false;
    vacrel->consider_bypass_optimization = true;

    /*
     * The index_cleanup param either disables index vacuuming and cleanup or
     * forces it to go ahead when we would otherwise apply the index bypass
     * optimization.  The default is 'auto', which leaves the final decision
     * up to lazy_vacuum().
     *
     * The truncate param allows user to avoid attempting relation truncation,
     * though it can't force truncation to happen.
     */
    Assert(params->index_cleanup != VACOPTVALUE_UNSPECIFIED);
    Assert(params->truncate != VACOPTVALUE_UNSPECIFIED &&
           params->truncate != VACOPTVALUE_AUTO);
    vacrel->do_index_vacuuming = true;
    vacrel->do_index_cleanup = true;
    vacrel->do_rel_truncate = (params->truncate != VACOPTVALUE_DISABLED);
    if (params->index_cleanup == VACOPTVALUE_DISABLED)
    {
        /* Force disable index vacuuming up-front */
        vacrel->do_index_vacuuming = false;
        vacrel->do_index_cleanup = false;
    }
    else if (params->index_cleanup == VACOPTVALUE_ENABLED)
    {
        /* Force index vacuuming.  Note that failsafe can still bypass. */
        vacrel->consider_bypass_optimization = false;
    }
    else
    {
        /* Default/auto, make all decisions dynamically */
        Assert(params->index_cleanup == VACOPTVALUE_AUTO);
    }

    vacrel->bstrategy = bstrategy;
    vacrel->old_rel_pages = rel->rd_rel->relpages;
    vacrel->old_live_tuples = rel->rd_rel->reltuples;
    vacrel->relfrozenxid = rel->rd_rel->relfrozenxid;
    vacrel->relminmxid = rel->rd_rel->relminmxid;

    /* Set cutoffs for entire VACUUM */
    vacrel->OldestXmin = OldestXmin;
    vacrel->FreezeLimit = FreezeLimit;
    vacrel->MultiXactCutoff = MultiXactCutoff;

    vacrel->relnamespace = get_namespace_name(RelationGetNamespace(rel));
    vacrel->relname = pstrdup(RelationGetRelationName(rel));
    vacrel->indname = NULL;
    vacrel->phase = VACUUM_ERRCB_PHASE_UNKNOWN;
…

       lazy_scan_heap是lazy vacuum的核心函数，该函数将首先扫描表（会用到vm文件），找到无效的元组和具有空闲空间的page，然后计算表的有效元组数，最后执行表和索引的清理操作。

    /* Do the vacuuming，核心函数 */
    lazy_scan_heap(vacrel, params, aggressive);

• 关闭表索引
• 计算实际是否扫描了所有未冻结页（aggressive模式），并设置scanned_all_unfrozen的值
• lazy_truncate_heap进行文件末尾的页截断（可选操作），这部分空间可以释放回操作系统。注意这个函数会短暂加8级锁，有可能影响业务

• ConditionalLockRelation(vacrel->rel, AccessExclusiveLock)

• 更新pg_class中的统计信息
• 清理vacrel中的索引统计信息及索引名

/* Done with indexes，关闭表索引 */
    vac_close_indexes(vacrel->nindexes, vacrel->indrels, NoLock);

    /*
     * Compute whether we actually scanned the all unfrozen pages. If we did,
     * we can adjust relfrozenxid and relminmxid.
     *
     * NB: We need to check this before truncating the relation, because that
     * will change ->rel_pages.
     */
    if ((vacrel->scanned_pages + vacrel->frozenskipped_pages)
        < vacrel->rel_pages)
    {
        Assert(!aggressive);
        scanned_all_unfrozen = false;
    }
    else
        scanned_all_unfrozen = true;

    /*
     * Optionally truncate the relation.尝试truncate文件末的页
     */
    if (should_attempt_truncation(vacrel))
    {
        /*
         * Update error traceback information.  This is the last phase during
         * which we add context information to errors, so we don't need to
         * revert to the previous phase.
         */
        update_vacuum_error_info(vacrel, NULL, VACUUM_ERRCB_PHASE_TRUNCATE,
                                 vacrel->nonempty_pages,
                                 InvalidOffsetNumber);
        lazy_truncate_heap(vacrel);
    }

    /*
     * Update statistics in pg_class.
     */
    new_rel_pages = vacrel->rel_pages;
    new_live_tuples = vacrel->new_live_tuples;

    visibilitymap_count(rel, &new_rel_allvisible, NULL);
    if (new_rel_allvisible > new_rel_pages)
        new_rel_allvisible = new_rel_pages;

    new_frozen_xid = scanned_all_unfrozen ? FreezeLimit : InvalidTransactionId;
    new_min_multi = scanned_all_unfrozen ? MultiXactCutoff : InvalidMultiXactId;

    vac_update_relstats(rel,
                        new_rel_pages,
                        new_live_tuples,
                        new_rel_allvisible,
                        vacrel->nindexes > 0,
                        new_frozen_xid,
                        new_min_multi,
                        false);
    …
    /* Cleanup index statistics and index names */
    for (int i = 0; i < vacrel->nindexes; i++)
    {
        if (vacrel->indstats[i])
            pfree(vacrel->indstats[i]);

        if (indnames && indnames[i])
            pfree(indnames[i]);
    }
}

如你所见，我们又掉进了新的坑里——lazy_scan_heap，后面继续研究研究这个函数~

参考：

《PostgreSQL数据库内核分析》

PostgreSQL 源码解读（128）- MVCC#12（vacuum过程-heap_vacuum_rel函数）_ITPUB博客

http://blog.itpub.net/6906/viewspace-2564641/

Postgresql Freezing 实现原理_13446560的技术博客_51CTO博客

https://www.pudn.com/news/6277722b517cd20ea491bf39.html