redis之listpack

最新推荐文章于 2024-09-16 10:23:45 发布

Orange_pa

最新推荐文章于 2024-09-16 10:23:45 发布

阅读量261

点赞数 1

文章标签： redis 数据库 c语言

本文链接：https://blog.csdn.net/Orange_pa/article/details/127847964

版权

listpack

listpack 四个参数

总字节数元素数量 Node节点结尾标识

在这里插入图片描述
listpack entry 节点结构
encoding 定义该元素的编码类型，会对不同长度的整数和字符串进行编码
data 实际存放的数据
len，encoding+data的总长度

listpackentry节点存储的信息

sval不为空 slen代表string长度
sval为空 lval保存值

/* Each entry in the listpack is either a string or an integer. */
typedef struct {
    /* When string is used, it is provided with the length (slen). */
    unsigned char *sval;
    uint32_t slen;
    /* When integer is used, 'sval' is NULL, and lval holds the value. */
    long long lval;
} listpackEntry;

listpackEntry中的len记录的是当前entry的长度，而非上一个entry的长度。

宏定义

#define LP_HDR_SIZE 6       /* 32 bit total len + 16 bit number of elements. */
#define LP_EOF 0xFF

hdr中长度为6字节，申请堆内存时共申请LP_HDR_SIZE +1，4字节用来记录totalLen，2字节用来记录元素的个数，+1的一个字节用来标识end，end也恒为0xFF

插入的宏定义

插入有三种模式

/* lpInsert() where argument possible values: */
#define LP_BEFORE 0
#define LP_AFTER 1
#define LP_REPLACE 2

增删改都是做统一处理
整体上 listpack的主要改进是为了避免连锁更新，(对比ziplist)

encoding内容

encoding:同样是为了提高内存使用率,reids也对encoding经过了复杂的设计.

编码	内容
0xxxxxxx	表示非负小整数，可以表示 0~127。
10xxxxxx	表示小字符串，长度范围是 0~63，content 字段为字符串的内容。
110xxxxx	yyyyyyyy 表示有符号整数，范围是-2048~2047。
1110xxxx	yyyyyyyy 表示中等长度的字符串，长度范围是 0~2047，content 字段为字符串的内容。
11110000	aaaaaaaa bbbbbbbb cccccccc dddddddd 表示大字符串，四个字节表示长度,content 字段为字符串内容。
11110001	aaaaaaaa bbbbbbbb 表示 2 字节有符号整数。内容保存在content中
11110010	aaaaaaaa bbbbbbbb cccccccc 表示 3 字节有符号整数。直接保存在encoding中
11110011	aaaaaaaa bbbbbbbb cccccccc dddddddd 表示 4 字节有符号整数。直接保存在encoding中
11110011	aaaaaaaa … hhhhhhhh 表示 8 字节有符号整数。直接保存在encoding中
11111111	表示 listpack 的结束符号，也就是 0xFF

部分函数功能

函数名	功能
unsigned char lpSeek(unsigned char lp, long index)	获取到lp的数量和index查找的节点位置，找打udp前半截就从左到右遍历，后半截数据位则反之，一直等到找到对应的节点后再返回对应节点值。
int lpValidateNext(unsigned char lp, unsigned char *pp, size_t lpbytes)	验证单个listpack节点的完整性，并跳转至下一个首先判断有没有超限，然后解析头部，获取长度，判断加上数据部分是不是也超限；获取节点长度和需要用的编码数，跳转下一节点，确保尾部编码长度和首部的契合
int lpValidateIntegrity(unsigned char lp, size_t size, int deep, listpackValidateEntryCB entry_cb, void cb_userdata)	从头部、长度、尾部标识、每一个节点的有效性和数据的吻合性来验证数据的可靠性
void lpRepr(unsigned char *lp)	打印节点

/* Create a new, empty listpack.
 * On success the new listpack is returned, otherwise an error is returned.
 * Pre-allocate at least `capacity` bytes of memory,
 * over-allocated memory can be shrunk by `lpShrinkToFit`.
 * */
unsigned char *lpNew(size_t capacity) {
    unsigned char *lp = lp_malloc(capacity > LP_HDR_SIZE+1 ? capacity : LP_HDR_SIZE+1);
    if (lp == NULL) return NULL;
    lpSetTotalBytes(lp,LP_HDR_SIZE+1);
    lpSetNumElements(lp,0);
    lp[LP_HDR_SIZE] = LP_EOF;
    return lp;
}

在这里插入图片描述

关键函数解析

其实最不同的就是insert
所以针对这几个来说是

insert函数源码与解析

/* Insert, delete or replace the specified string element 'elestr' of length
 * 'size' or integer element 'eleint' at the specified position 'p', with 'p'
 * being a listpack element pointer obtained with lpFirst(), lpLast(), lpNext(),
 * lpPrev() or lpSeek().
 *
 * The element is inserted before, after, or replaces the element pointed
 * by 'p' depending on the 'where' argument, that can be LP_BEFORE, LP_AFTER
 * or LP_REPLACE.
 * 
 * If both 'elestr' and `eleint` are NULL, the function removes the element
 * pointed by 'p' instead of inserting one.
 * If `eleint` is non-NULL, 'size' is the length of 'eleint', the function insert
 * or replace with a 64 bit integer, which is stored in the 'eleint' buffer.
 * If 'elestr` is non-NULL, 'size' is the length of 'elestr', the function insert
 * or replace with a string, which is stored in the 'elestr' buffer.
 * 
 * Returns NULL on out of memory or when the listpack total length would exceed
 * the max allowed size of 2^32-1, otherwise the new pointer to the listpack
 * holding the new element is returned (and the old pointer passed is no longer
 * considered valid)
 *
 * If 'newp' is not NULL, at the end of a successful call '*newp' will be set
 * to the address of the element just added, so that it will be possible to
 * continue an interaction with lpNext() and lpPrev().
 *
 * For deletion operations (both 'elestr' and 'eleint' set to NULL) 'newp' is
 * set to the next element, on the right of the deleted one, or to NULL if the
 * deleted element was the last one. */
unsigned char *lpInsert(unsigned char *lp, unsigned char *elestr, unsigned char *eleint,
                        uint32_t size, unsigned char *p, int where, unsigned char **newp)
{
    unsigned char intenc[LP_MAX_INT_ENCODING_LEN];
    unsigned char backlen[LP_MAX_BACKLEN_SIZE];

    uint64_t enclen; /* The length of the encoded element. */
    int delete = (elestr == NULL && eleint == NULL);

    /* when deletion, it is conceptually replacing the element with a
     * zero-length element. So whatever we get passed as 'where', set
     * it to LP_REPLACE. */
    if (delete) where = LP_REPLACE;

    /* If we need to insert after the current element, we just jump to the
     * next element (that could be the EOF one) and handle the case of
     * inserting before. So the function will actually deal with just two
     * cases: LP_BEFORE and LP_REPLACE. */
    if (where == LP_AFTER) {
        p = lpSkip(p);
        where = LP_BEFORE;
        ASSERT_INTEGRITY(lp, p);
    }

    /* Store the offset of the element 'p', so that we can obtain its
     * address again after a reallocation. */
    unsigned long poff = p-lp;

    int enctype;
    if (elestr) {
        /* Calling lpEncodeGetType() results into the encoded version of the
        * element to be stored into 'intenc' in case it is representable as
        * an integer: in that case, the function returns LP_ENCODING_INT.
        * Otherwise if LP_ENCODING_STR is returned, we'll have to call
        * lpEncodeString() to actually write the encoded string on place later.
        *
        * Whatever the returned encoding is, 'enclen' is populated with the
        * length of the encoded element. */
        enctype = lpEncodeGetType(elestr,size,intenc,&enclen);
        if (enctype == LP_ENCODING_INT) eleint = intenc;
    } else if (eleint) {
        enctype = LP_ENCODING_INT;
        enclen = size; /* 'size' is the length of the encoded integer element. */
    } else {
        enctype = -1;
        enclen = 0;
    }

    /* We need to also encode the backward-parsable length of the element
     * and append it to the end: this allows to traverse the listpack from
     * the end to the start. */
     //通过上边提前取得的enclen，解码length的长度，如果为删除操作则无需
    unsigned long backlen_size = (!delete) ? lpEncodeBacklen(backlen,enclen) : 0;
    uint64_t old_listpack_bytes = lpGetTotalBytes(lp);
    uint32_t replaced_len  = 0;
     //这里是处理删除操作的具体方法
    if (where == LP_REPLACE) {
        replaced_len = lpCurrentEncodedSizeUnsafe(p);
        replaced_len += lpEncodeBacklen(NULL,replaced_len);
        ASSERT_INTEGRITY_LEN(lp, p, replaced_len);
    }
 //在这里就可以获取到现在的lp大小了
    uint64_t new_listpack_bytes = old_listpack_bytes + enclen + backlen_size
                                  - replaced_len;
//越界判断
    if (new_listpack_bytes > UINT32_MAX) return NULL;

    /* We now need to reallocate in order to make space or shrink the
     * allocation (in case 'when' value is LP_REPLACE and the new element is
     * smaller). However we do that before memmoving the memory to
     * make room for the new element if the final allocation will get
     * larger, or we do it after if the final allocation will get smaller. */
//poff为之前获取到的p之前的位置，这里的dst则定位到该在哪个地方中进行插入或删除了
    unsigned char *dst = lp + poff; /* May be updated after reallocation. */

    /* Realloc before: we need more room. */
    if (new_listpack_bytes > old_listpack_bytes &&
        new_listpack_bytes > lp_malloc_size(lp)) {
        if ((lp = lp_realloc(lp,new_listpack_bytes)) == NULL) return NULL;
        dst = lp + poff;
    }

    /* Setup the listpack relocating the elements to make the exact room
     * we need to store the new one. */
   //本函数中只有LP_BEFORE与LP_REPLACE
    if (where == LP_BEFORE) {
        memmove(dst+enclen+backlen_size,dst,old_listpack_bytes-poff);
    } else { /* LP_REPLACE. */
     //发现删除就是把当前元素len置空
        memmove(dst+enclen+backlen_size,
                dst+replaced_len,
                old_listpack_bytes-poff-replaced_len);
    }

    /* Realloc after: we need to free space. */
    if (new_listpack_bytes < old_listpack_bytes) {
        if ((lp = lp_realloc(lp,new_listpack_bytes)) == NULL) return NULL;
        dst = lp + poff;
    }

    /* Store the entry. */
    if (newp) {
        *newp = dst;
        /* In case of deletion, set 'newp' to NULL if the next element is
         * the EOF element. */
        if (delete && dst[0] == LP_EOF) *newp = NULL;
    }
    if (!delete) {
        if (enctype == LP_ENCODING_INT) {
            memcpy(dst,eleint,enclen);
        } else if (elestr) {
            lpEncodeString(dst,elestr,size);
        } else {
            redis_unreachable();
        }
        dst += enclen;
        memcpy(dst,backlen,backlen_size);
        dst += backlen_size;
    }
    //更新hd
    /* Update header. */
    if (where != LP_REPLACE || delete) {
        uint32_t num_elements = lpGetNumElements(lp);
        if (num_elements != LP_HDR_NUMELE_UNKNOWN) {
            if (!delete)
                lpSetNumElements(lp,num_elements+1);
            else
                lpSetNumElements(lp,num_elements-1);
        }
    }
    lpSetTotalBytes(lp,new_listpack_bytes);

#if 0
    /* This code path is normally disabled: what it does is to force listpack
     * to return *always* a new pointer after performing some modification to
     * the listpack, even if the previous allocation was enough. This is useful
     * in order to spot bugs in code using listpacks: by doing so we can find
     * if the caller forgets to set the new pointer where the listpack reference
     * is stored, after an update. */
    unsigned char *oldlp = lp;
    lp = lp_malloc(new_listpack_bytes);
    memcpy(lp,oldlp,new_listpack_bytes);
    if (newp) {
        unsigned long offset = (*newp)-oldlp;
        *newp = lp + offset;
    }
    /* Make sure the old allocation contains garbage. */
    memset(oldlp,'A',new_listpack_bytes);
    lp_free(oldlp);
#endif

    return lp;
}

注释部分

参数：unsigned char *lp, unsigned char *elestr, unsigned char *eleint,
uint32_t size, unsigned char *p, int where, unsigned char **newp

该元素被插入到’p’指向的元素之前、之后或替换，取决于’where’参数，该参数可以是LP_BEFORE、LP_AFTER或LP_REPLACE。（见前面宏定义）

如果’elestr’和’eleint '都为NULL，函数将删除由’p’指向的元素，而不是插入一个。
如果’eleint’非null， 'size’是’eleint’的长度，函数插入或替换为64位整数，存储在’eleint’缓冲区中。
如果’elestr’非null， 'size’是’elestr’的长度，‘elestr’是函数插入或替换为字符串，存储在’elestr’缓冲区中。
当内存不足或当listpack总长度超过允许的最大大小2^32-1时，返回NULL，否则返回指向包含新元素的listpack的新指针(并且传递的旧指针不再被认为有效)
如果’newp’不是NULL，在成功调用的末尾’*newp’将被设置为刚刚添加的元素的地址，这样就有可能继续与lpNext()和lpPrev()的交互。
对于删除操作('elestr’和’eleint’都设置为NULL)'newp’被设置为下一个元素，在被删除元素的右边，如果被删除的元素是最后一个，则设置为NULL。

如果’elestr’和’eleint '都为NULL，delete=1，where = LP_REPLACE

    int delete = (elestr == NULL && eleint == NULL);
    /* when deletion, it is conceptually replacing the element with a
     * zero-length element. So whatever we get passed as 'where', set
     * it to LP_REPLACE. */
    if (delete) where = LP_REPLACE;

如果where为LP_AFTER，获取下一个节点，在这个节点之前加入
把后插统一为前插操作

    if (where == LP_AFTER) {
        p = lpSkip(p);
        where = LP_BEFORE;
        ASSERT_INTEGRITY(lp, p);
    }

解析enctype标识类型和enclen标识长度在这里插入图片描述

其他函数

释放函数

/* Free the specified listpack. */
void lpFree(unsigned char *lp) {
    lp_free(lp);
}

重新分配空间

/* Shrink the memory to fit. */
##lp_realloc 就是 zrealloc
unsigned char* lpShrinkToFit(unsigned char *lp) {
    size_t size = lpGetTotalBytes(lp);
    if (size < lp_malloc_size(lp)) {
        return lp_realloc(lp, size);
    } else {
        return lp;
    }
}