【redis源码学习】redis 中的“消息队列” Stream，三面蚂蚁金服成功拿到offer

先自我介绍一下，小编浙江大学毕业，去过华为、字节跳动等大厂，目前阿里P7

深知大多数程序员，想要提升技能，往往是自己摸索成长，但自己不成体系的自学效果低效又漫长，而且极易碰到天花板技术停滞不前！

因此收集整理了一份《2024年最新Java开发全套学习资料》，初衷也很简单，就是希望能够帮助到想自学提升又不知道该从何学起的朋友。

既有适合小白学习的零基础资料，也有适合3年以上经验的小伙伴深入学习提升的进阶课程，涵盖了95%以上Java开发知识点，真正体系化！

由于文件比较多，这里只是将部分目录截图出来，全套包含大厂面经、学习笔记、源码讲义、实战项目、大纲路线、讲解视频，并且后续会持续更新

如果你需要这些资料，可以添加V获取：vip1024b （备注Java）

正文

group. Consumers that will just ask for more

messages will served with IDs > than this. */

rax pel; / Pending entries list. This is a radix tree that

has every message delivered to consumers (without

the NOACK option) that was yet not acknowledged

as processed. The key of the radix tree is the

ID as a 64 bit big endian number, while the

associated value is a streamNACK structure.*/

rax consumers; / A radix tree representing the consumers by name

and their associated representation in the form

of streamConsumer structures. */

} streamCG;

pel：组中所有待确认信息

consumers：组中所有消费者，Rax值指向 streamConsumer 结构体。

---

/* A specific consumer in a consumer group. */

typedef struct streamConsumer {

mstime_t seen_time; /* Last time this consumer was active. */

sds name; /* Consumer name. This is how the consumer

will be identified in the consumer group

protocol. Case sensitive. */

rax pel; / Consumer specific pending entries list: all

the pending messages delivered to this

consumer not yet acknowledged. Keys are

big endian message IDs, while values are

the same streamNACK structure referenced

in the “pel” of the conumser group structure

itself, so the value is shared. */

} streamConsumer;

---

这两处的 pel 都指向待确认的消息信息，如下结构体保存这些信息：

/* Pending (yet not acknowledged) message in a consumer group. */

typedef struct streamNACK {

mstime_t delivery_time; /* Last time this message was delivered. */

uint64_t delivery_count; /* Number of times this message was delivered.*/

streamConsumer consumer; / The consumer this message was delivered to

in the last delivery. */

} streamNACK;

---

至此，整个 Stream 的脉络已经明朗。再看一个迭代器吧。

像这样一个有点麻烦的结构，那肯定得定制一个迭代器啊！！！

而且这个迭代器，不简单。

/* We define an iterator to iterate stream items in an abstract way, without

• caring about the radix tree + listpack representation. Technically speaking

• the iterator is only used inside streamReplyWithRange(), so could just

• be implemented inside the function, but practically there is the AOF

• rewriting code that also needs to iterate the stream to emit the XADD

• commands. */

typedef struct streamIterator {

stream *stream; //当前迭代器正在遍历的消息流

streamID master_id; /* ID of the master entry at listpack head. */

uint64_t master_fields_count; /* Master entries # of fields. */

unsigned char master_fields_start; / Master entries start in listpack. */

unsigned char master_fields_ptr; / Master field to emit next. */

int entry_flags; /* Flags of entry we are emitting. */

int rev; /* True if iterating end to start (reverse). */

uint64_t start_key[2]; /* Start key as 128 bit big endian. */

uint64_t end_key[2]; /* End key as 128 bit big endian. */

raxIterator ri; /* Rax iterator. */

unsigned char lp; / Current listpack. */

unsigned char lp_ele; / Current listpack cursor. */

unsigned char lp_flags; / Current entry flags pointer. */

/* Buffers used to hold the string of lpGet() when the element is

• integer encoded, so that there is no string representation of the

• element inside the listpack itself. */

unsigned char field_buf[LP_INTBUF_SIZE];

unsigned char value_buf[LP_INTBUF_SIZE];

} streamIterator;

---

Stream 操作

---

添加消息

/* Adds a new item into the stream ‘s’ having the specified number of

• field-value pairs as specified in ‘numfields’ and stored into ‘argv’.

• Returns the new entry ID populating the ‘added_id’ structure.

• If ‘use_id’ is not NULL, the ID is not auto-generated by the function,

• but instead the passed ID is used to add the new entry. In this case

• adding the entry may fail as specified later in this comment.

• The function returns C_OK if the item was added, this is always true

• if the ID was generated by the function. However the function may return

• C_ERR if an ID was given via ‘use_id’, but adding it failed since the

• current top ID is greater or equal. */

int streamAppendItem(stream *s, robj **argv, int64_t numfields, streamID *added_id, streamID *use_id) {

/* Generate the new entry ID. */

streamID id;

if (use_id)

id = *use_id;

else

streamNextID(&s->last_id,&id);

/* Check that the new ID is greater than the last entry ID

• or return an error. Automatically generated IDs might

• overflow (and wrap-around) when incrementing the sequence

part. */

if (streamCompareID(&id,&s->last_id) <= 0) return C_ERR;

/* Add the new entry. */

raxIterator ri;

raxStart(&ri,s->rax);

raxSeek(&ri,“$”,NULL,0);

size_t lp_bytes = 0; /* Total bytes in the tail listpack. */

unsigned char lp = NULL; / Tail listpack pointer. */

/* Get a reference to the tail node listpack. */

if (raxNext(&ri)) {

lp = ri.data;

lp_bytes = lpBytes(lp);

}

raxStop(&ri);

/* We have to add the key into the radix tree in lexicographic order,

• to do so we consider the ID as a single 128 bit number written in

• big endian, so that the most significant bytes are the first ones. */

uint64_t rax_key[2]; /* Key in the radix tree containing the listpack.*/

streamID master_id; /* ID of the master entry in the listpack. */

/* Create a new listpack and radix tree node if needed. Note that when

• a new listpack is created, we populate it with a “master entry”. This

• is just a set of fields that is taken as references in order to compress

• the stream entries that we’ll add inside the listpack.

• Note that while we use the first added entry fields to create

• the master entry, the first added entry is NOT represented in the master

• entry, which is a stand alone object. But of course, the first entry

• will compress well because it’s used as reference.

• The master entry is composed like in the following example:

• ±------±--------±-----------±--------±-/–±--------±--------±+

• | count | deleted | num-fields | field_1 | field_2 | … | field_N |0|

• ±------±--------±-----------±--------±-/–±--------±--------±+

• count and deleted just represent respectively the total number of

• entries inside the listpack that are valid, and marked as deleted

• (deleted flag in the entry flags set). So the total number of items

• actually inside the listpack (both deleted and not) is count+deleted.

• The real entries will be encoded with an ID that is just the

• millisecond and sequence difference compared to the key stored at

• the radix tree node containing the listpack (delta encoding), and

• if the fields of the entry are the same as the master entry fields, the

• entry flags will specify this fact and the entry fields and number

• of fields will be omitted (see later in the code of this function).

• The “0” entry at the end is the same as the ‘lp-count’ entry in the

• regular stream entries (see below), and marks the fact that there are

• no more entries, when we scan the stream from right to left. */

/* First of all, check if we can append to the current macro node or

• if we need to switch to the next one. ‘lp’ will be set to NULL if

• the current node is full. */

if (lp != NULL) {

if (server.stream_node_max_bytes &&

lp_bytes >= server.stream_node_max_bytes)

{

lp = NULL;

} else if (server.stream_node_max_entries) {

int64_t count = lpGetInteger(lpFirst(lp));

if (count >= server.stream_node_max_entries) lp = NULL;

}

}

int flags = STREAM_ITEM_FLAG_NONE;

if (lp == NULL || lp_bytes >= server.stream_node_max_bytes) {

master_id = id;

streamEncodeID(rax_key,&id);

/* Create the listpack having the master entry ID and fields. */

lp = lpNew();

lp = lpAppendInteger(lp,1); /* One item, the one we are adding. */

lp = lpAppendInteger(lp,0); /* Zero deleted so far. */

lp = lpAppendInteger(lp,numfields);

for (int64_t i = 0; i < numfields; i++) {

sds field = argv[i*2]->ptr;

lp = lpAppend(lp,(unsigned char*)field,sdslen(field));

}

lp = lpAppendInteger(lp,0); /* Master entry zero terminator. */

raxInsert(s->rax,(unsigned char*)&rax_key,sizeof(rax_key),lp,NULL);

/* The first entry we insert, has obviously the same fields of the

• master entry. */

flags |= STREAM_ITEM_FLAG_SAMEFIELDS;

} else {

serverAssert(ri.key_len == sizeof(rax_key));

memcpy(rax_key,ri.key,sizeof(rax_key));

/* Read the master ID from the radix tree key. */

streamDecodeID(rax_key,&master_id);

unsigned char *lp_ele = lpFirst(lp);

/* Update count and skip the deleted fields. */

int64_t count = lpGetInteger(lp_ele);

lp = lpReplaceInteger(lp,&lp_ele,count+1);

lp_ele = lpNext(lp,lp_ele); /* seek deleted. */

lp_ele = lpNext(lp,lp_ele); /* seek master entry num fields. */

/* Check if the entry we are adding, have the same fields

• as the master entry. */

int64_t master_fields_count = lpGetInteger(lp_ele);

lp_ele = lpNext(lp,lp_ele);

if (numfields == master_fields_count) {

int64_t i;

for (i = 0; i < master_fields_count; i++) {

sds field = argv[i*2]->ptr;

int64_t e_len;

unsigned char buf[LP_INTBUF_SIZE];

unsigned char *e = lpGet(lp_ele,&e_len,buf);

/* Stop if there is a mismatch. */

if (sdslen(field) != (size_t)e_len ||

memcmp(e,field,e_len) != 0) break;

lp_ele = lpNext(lp,lp_ele);

}

/* All fields are the same! We can compress the field names

• setting a single bit in the flags. */

if (i == master_fields_count) flags |= STREAM_ITEM_FLAG_SAMEFIELDS;

}

}

/* Populate the listpack with the new entry. We use the following

• encoding:

• ±----±-------±---------±------±------±/-±------±------±-------+

• |flags|entry-id|num-fields|field-1|value-1|…|field-N|value-N|lp-count|

• ±----±-------±---------±------±------±/-±------±------±-------+

• However if the SAMEFIELD flag is set, we have just to populate

• the entry with the values, so it becomes:

• ±----±-------±------±/-±------±-------+

• |flags|entry-id|value-1|…|value-N|lp-count|

• ±----±-------±------±/-±------±-------+

• The entry-id field is actually two separated fields: the ms

• and seq difference compared to the master entry.

• The lp-count field is a number that states the number of listpack pieces

• that compose the entry, so that it’s possible to travel the entry

• in reverse order: we can just start from the end of the listpack, read

• the entry, and jump back N times to seek the “flags” field to read

• the stream full entry. */

lp = lpAppendInteger(lp,flags);

lp = lpAppendInteger(lp,id.ms - master_id.ms);

lp = lpAppendInteger(lp,id.seq - master_id.seq);

if (!(flags & STREAM_ITEM_FLAG_SAMEFIELDS))

lp = lpAppendInteger(lp,numfields);

for (int64_t i = 0; i < numfields; i++) {

sds field = argv[i2]->ptr, value = argv[i2+1]->ptr;

if (!(flags & STREAM_ITEM_FLAG_SAMEFIELDS))

lp = lpAppend(lp,(unsigned char*)field,sdslen(field));

lp = lpAppend(lp,(unsigned char*)value,sdslen(value));

}

/* Compute and store the lp-count field. */

int64_t lp_count = numfields;

lp_count += 3; /* Add the 3 fixed fields flags + ms-diff + seq-diff. */

if (!(flags & STREAM_ITEM_FLAG_SAMEFIELDS)) {

/* If the item is not compressed, it also has the fields other than

• the values, and an additional num-fileds field. */

lp_count += numfields+1;

}

lp = lpAppendInteger(lp,lp_count);

/* Insert back into the tree in order to update the listpack pointer. */

if (ri.data != lp)

raxInsert(s->rax,(unsigned char*)&rax_key,sizeof(rax_key),lp,NULL);

s->length++;

s->last_id = id;

if (added_id) *added_id = id;

return C_OK;

}

流程简析：

1、获取rax最后一个key所在的节点，由于Rax树是按照消息id的顺序存储的，所以最后一个key节点存储了上一次插入的消息。

2、查看该节点是否可以插入这条新的消息。

3、如果该节点已经不能再插入新的消息（listpack为空或已经到达最大存储值），初始化新建的listpack；如果还可以用，则对比插入的消息与listpack中master消息对应的fields内容是否完全一致，完全一致则表明该消息可以复用master的field。

4、将待插入的消息内容插入到新建的listpack中或者原来的rax的最后一个key节点对应的listpack中。

---

新增消费组

/* Create a new consumer group in the context of the stream ‘s’, having the

• specified name and last server ID. If a consumer group with the same name

• already existed NULL is returned, otherwise the pointer to the consumer

• group is returned. */

streamCG *streamCreateCG(stream *s, char *name, size_t namelen, streamID *id) {

if (s->cgroups == NULL) s->cgroups = raxNew();

if (raxFind(s->cgroups,(unsigned char*)name,namelen) != raxNotFound)

return NULL;

streamCG *cg = zmalloc(sizeof(*cg));

cg->pel = raxNew();

cg->consumers = raxNew();

cg->last_id = *id;

raxInsert(s->cgroups,(unsigned char*)name,namelen,cg,NULL);

return cg;

}

为消息流新增一个消费组，以消费组的名称为key，该消费组的streamCG结构为value，放入rax中。

---

删除消息

/* Remove the current entry from the stream: can be called after the

• GetID() API or after any GetField() call, however we need to iterate

• a valid entry while calling this function. Moreover the function

• requires the entry ID we are currently iterating, that was previously

• returned by GetID().

• Note that after calling this function, next calls to GetField() can’t

• be performed: the entry is now deleted. Instead the iterator will

• automatically re-seek to the next entry, so the caller should continue

• with GetID(). */

void streamIteratorRemoveEntry(streamIterator *si, streamID *current) {

unsigned char *lp = si->lp;

int64_t aux;

/* We do not really delete the entry here. Instead we mark it as

• deleted flagging it, and also incrementing the count of the

• deleted entries in the listpack header.

• We start flagging: */

int flags = lpGetInteger(si->lp_flags);

flags |= STREAM_ITEM_FLAG_DELETED;

lp = lpReplaceInteger(lp,&si->lp_flags,flags);

/* Change the valid/deleted entries count in the master entry. */

unsigned char *p = lpFirst(lp);

aux = lpGetInteger§;

if (aux == 1) {

/* If this is the last element in the listpack, we can remove the whole

• node. */

lpFree(lp);

raxRemove(si->stream->rax,si->ri.key,si->ri.key_len,NULL);

} else {

/* In the base case we alter the counters of valid/deleted entries. */

lp = lpReplaceInteger(lp,&p,aux-1);

p = lpNext(lp,p); /* Seek deleted field. */

aux = lpGetInteger§;

lp = lpReplaceInteger(lp,&p,aux+1);

/* Update the listpack with the new pointer. */

if (si->lp != lp)

raxInsert(si->stream->rax,si->ri.key,si->ri.key_len,lp,NULL);

}

/* Update the number of entries counter. */

si->stream->length–;

/* Re-seek the iterator to fix the now messed up state. */

streamID start, end;

if (si->rev) {

streamDecodeID(si->start_key,&start);

end = *current;

} else {

start = *current;

streamDecodeID(si->end_key,&end);

最后

本人也收藏了一份Java面试核心知识点来应付面试，借着这次机会可以送给我的读者朋友们:

目录：

Java面试核心知识点

一共有30个专题，足够读者朋友们应付面试啦，也节省朋友们去到处搜刮资料自己整理的时间！

Java面试核心知识点

网上学习资料一大堆，但如果学到的知识不成体系，遇到问题时只是浅尝辄止，不再深入研究，那么很难做到真正的技术提升。

需要这份系统化的资料的朋友，可以添加V获取：vip1024b （备注Java）

一个人可以走的很快，但一群人才能走的更远！不论你是正从事IT行业的老鸟或是对IT行业感兴趣的新人，都欢迎加入我们的的圈子（技术交流、学习资源、职场吐槽、大厂内推、面试辅导），让我们一起学习成长！
len,lp,NULL);

}

/* Update the number of entries counter. */

si->stream->length–;

/* Re-seek the iterator to fix the now messed up state. */

streamID start, end;

if (si->rev) {

streamDecodeID(si->start_key,&start);

end = *current;

} else {

start = *current;

streamDecodeID(si->end_key,&end);

最后

本人也收藏了一份Java面试核心知识点来应付面试，借着这次机会可以送给我的读者朋友们:

目录：

[外链图片转存中…(img-NRlVMNEP-1713438253533)]

Java面试核心知识点

一共有30个专题，足够读者朋友们应付面试啦，也节省朋友们去到处搜刮资料自己整理的时间！

[外链图片转存中…(img-3GMNV95M-1713438253533)]

Java面试核心知识点

网上学习资料一大堆，但如果学到的知识不成体系，遇到问题时只是浅尝辄止，不再深入研究，那么很难做到真正的技术提升。

需要这份系统化的资料的朋友，可以添加V获取：vip1024b （备注Java）
[外链图片转存中…(img-KjtlLCpS-1713438253533)]

一个人可以走的很快，但一群人才能走的更远！不论你是正从事IT行业的老鸟或是对IT行业感兴趣的新人，都欢迎加入我们的的圈子（技术交流、学习资源、职场吐槽、大厂内推、面试辅导），让我们一起学习成长！