Redis源码阅读(二)- stream与radix树

目录

stream介绍

stream操作命令

追加消息XADD

范围查询XRANGE，XREVRANGE

读取消息命令XREAD

 消费组模式相关命令

创建消费组XGROUP CREATE

 读取消费组消息XREADGROUP

全部命令

stream实现原理

radix查找和遍历

radix查找过程       

radix遍历

radix插入

radix删除

redis相关实现原理

radix相关结构定义

stream结构定义

listpack紧凑列表

消费组功能实现

总结

---

stream介绍

        stream是redis5.0引入的新数据类型，是消息队列的一种实现方式，实现了完整的消息队列功能。主要特点包括：

1. stream 提供了消息的持久化和主备复制功能，保证消息不会丢失。
2. 阻塞操作，允许消费者等待生产者向流中添加新数据。
3. 消费组功能（类似Kafka实现），允许一组客户端协作使用同一消息流的不同部分。

   

stream操作命令

追加消息XADD

XADD命令向stream中追加消息，如果stream不存在，则创建。消息格式：

XADD key ID field value [feild value ...]

key: stream名称，如果不存在则创建
ID： 消息ID，*表示redis自动创建ID
field value: 值，K-V格式

执行XADD命令插入内容，返回消息ID，消息ID包含两部分${时间戳}-${序列}

> XADD teststream * name redis version 6.0
"1641651260618-0"

执行XADD命令，指定消息ID

> XADD teststream 1641651260618-1 name java version 1.8
"1641651260618-1"

范围查询XRANGE，XREVRANGE

XRANGE命令实现范围查询，XREVRANGE逆序查询，命令格式：

XRANGE|XREVRANGE key start end [COUNT count]

key：straem名称
start：开始ID，- 表示最小
end：结束ID，+ 表示最大
count：返回数量

 命令演示

> XRANGE teststream - + 
1) 1) "1641651260618-0"
   2) 1) "name"
      2) "redis"
      3) "version"
      4) "6.0"
2) 1) "1641651260618-1"
   2) 1) "name"
      2) "java"
      3) "version"
      4) "1.8"

# 指定返回数量
> XRANGE teststream - + count 1
1) 1) "1641651260618-0"
   2) 1) "name"
      2) "redis"
      3) "version"
      4) "6.0"

# 指定时间参数
> XRANGE teststream 1641651260618 + 
1) 1) "1641651260618-0"
   2) 1) "name"
      2) "redis"
      3) "version"
      4) "6.0"
2) 1) "1641651260618-1"
   2) 1) "name"
      2) "java"
      3) "version"
      4) "1.8"

# 指定开始结束ID参数
> XRANGE teststream 1641651260618-0 1641651260618-0 
1) 1) "1641651260618-0"
   2) 1) "name"
      2) "redis"
      3) "version"
      4) "6.0"

# XREVRANGE命令逆序遍历
> XREVRANGE teststream + -
1) 1) "1641651260618-1"
   2) 1) "name"
      2) "java"
      3) "version"
      4) "1.8"
2) 1) "1641651260618-0"
   2) 1) "name"
      2) "redis"
      3) "version"
      4) "6.0"

读取消息命令XREAD

XREAD命令读取消息，可以阻塞或非阻塞，命令格式：

XREAD [COUNT count] [BLOCK milliseconds] STREAMS key [key ...] ID [ID ...]

COUNT ：数量
BLOCK ：可选，阻塞毫秒数，没有设置就是非阻塞模式
key ：stream名称
ID ：消息 ID，$表示stream中最大的ID，即读取最新的消息

XREAD命令演示，非阻塞方式

> XREAD STREAMS teststream 0
1) 1) "teststream"
   2) 1) 1) "1641651260618-0"
         2) 1) "name"
            2) "redis"
            3) "version"
            4) "6.0"
      2) 1) "1641651260618-1"
         2) 1) "name"
            2) "java"
            3) "version"
            4) "1.8"

XREAD命令演示，阻塞方式

> XREAD BLOCK 1000000 STREAMS teststream $
1) 1) "teststream"
   2) 1) 1) "1641653381688-0"
         2) 1) "name"
            2) "mysql"
            3) "version"
            4) "8.0"
(29.36s)

# 另开一个终端，push消息
> xadd teststream * name mysql version 8.0
"1641653381688-0"

 消费组模式相关命令

创建消费组XGROUP CREATE

XGROUP [CREATE key groupname ID|$ [MKSTREAM]] [SETID key groupname ID|$] [DESTROY key groupname] [CREATECONSUMER key groupname consume

key ：stream名称，如果不存在就创建
groupname ：消费组名
ID ： 指定开始消费ID，或使用 $表示从尾部开始消费，只接受新消息，当前 Stream 消息会全部忽略

 命令演示

# 创建消费组，从头开始消费
> XGROUP CREATE teststream testgroup1 0
OK

# 创建消费组，消费最新消息
> XGROUP CREATE teststream testgroup2 $
OK

 读取消费组消息XREADGROUP

XREADGROUP指定消费组消费消息，与XREAD命令一样，XREADGROUP支持阻塞模式，命令格式：

XREADGROUP GROUP group consumer [COUNT count] [BLOCK milliseconds] [NOACK] STREAMS key [key ...] ID [ID ...]

group ：消费组名
consumer ：消费者名
COUNT ： 读取数量
BLOCK ： 阻塞毫秒数
key ： stream名称
ID ： 消息 ID，特殊ID > 表示读取未消费过的消息

命令演示

> XREADGROUP GROUP testgroup1 Alice STREAMS teststream >
1) 1) "teststream"
   2) 1) 1) "1641651260618-0"
         2) 1) "name"
            2) "redis"
            3) "version"
            4) "6.0"
      2) 1) "1641651260618-1"
         2) 1) "name"
            2) "java"
            3) "version"
            4) "1.8"
      3) 1) "1641653381688-0"
         2) 1) "name"
            2) "mysql"
            3) "version"
            4) "8.0"

全部命令

由于篇幅原因，这里不对全部命令一一介绍了，有需要的可以去redis官网自行查阅

命令	说明
XTRIM	对流进行修剪，限制长度，ID较小的项目可能会被删除
XDEL	删除消息
XLEN	获取消息长度
XRANGE | XREVRANGE	范围查询
XREAD	读取消息
消费组相关
XGROUP [         CREATE         SETID         DELCONSUMER         DESTROY ]	消费者相关操作命令
XREADGROUP GROUP	读取消费组中的消息
XACK	将消息标记为"已处理"
XPENDING	显示待处理消息的相关信息
XCLAIM	改变待处理消息的所有权
XINFO [         GROUPS         STREAM ]	查看流和消费者组的相关信息

stream实现原理

        stream使用radix（基数树）作为存储结构，radix是一树形结构，适合查找和存储，radix特点：

• 树由叶子结点和非叶子结点组成
• 同级结点按照顺序排列
• 结点的key是由从根节点到结点的所有key组合而成
• 如果结点为空且只有一个子结点，需要与子结点合并

radix查找和遍历

radix查找过程       

radix遍历

        radix树的遍历与二叉树类似，可以参考二叉树的遍历方式。

radix插入

        radix树插入的各种场景       

radix删除

        删除是插入的逆操作，也需要区分场景，这里不详细介绍。

redis相关实现原理

radix相关结构定义

        rax结构定义radix树结构，结点中raxNode.data保存数据，包含key和value。

//radix根结点定义
typedef struct rax {
    raxNode *head;
    uint64_t numele;
    uint64_t numnodes;
} rax;

//
#define RAX_NODE_MAX_SIZE ((1<<29)-1)
typedef struct raxNode {
    uint32_t iskey:1;     /* Does this node contain a key? */
    uint32_t isnull:1;    /* Associated value is NULL (don't store it). */
    uint32_t iscompr:1;   /* Node is compressed. */
    uint32_t size:29;     /* Number of children, or compressed string len. */
    /* Data layout is as follows:
     *
     * If node is not compressed we have 'size' bytes, one for each children
     * character, and 'size' raxNode pointers, point to each child node.
     * Note how the character is not stored in the children but in the
     * edge of the parents:
     *
     * [header iscompr=0][abc][a-ptr][b-ptr][c-ptr](value-ptr?)
     *
     * if node is compressed (iscompr bit is 1) the node has 1 children.
     * In that case the 'size' bytes of the string stored immediately at
     * the start of the data section, represent a sequence of successive
     * nodes linked one after the other, for which only the last one in
     * the sequence is actually represented as a node, and pointed to by
     * the current compressed node.
     *
     * [header iscompr=1][xyz][z-ptr](value-ptr?)
     *
     * Both compressed and not compressed nodes can represent a key
     * with associated data in the radix tree at any level (not just terminal
     * nodes).
     *
     * If the node has an associated key (iskey=1) and is not NULL
     * (isnull=0), then after the raxNode pointers pointing to the
dhcccdcn     * in the representation above as "value-ptr" field).
     */
    unsigned char data[];
} raxNode;

stream结构定义

        stream相关结构定义，内部使用rax存储数据。    

/* Stream item ID: a 128 bit number composed of a milliseconds time and
 * a sequence counter. IDs generated in the same millisecond (or in a past
 * millisecond if the clock jumped backward) will use the millisecond time
 * of the latest generated ID and an incremented sequence. */
typedef struct streamID {
    uint64_t ms;        /* Unix time in milliseconds. */
    uint64_t seq;       /* Sequence number. */
} streamID;

typedef struct stream {
    rax *rax;               /* The radix tree holding the stream. */
    uint64_t length;        /* Number of elements inside this stream. */
    streamID last_id;       /* Zero if there are yet no items. */
    rax *cgroups;           /* Consumer groups dictionary: name -> streamCG */
} stream;

/* We define an iterator to iterate stream items in an abstract way, without
 * caring about the radix tree + listpack representation. Technically speaking
 * the iterator is only used inside streamReplyWithRange(), so could just
 * be implemented inside the function, but practically there is the AOF
 * rewriting code that also needs to iterate the stream to emit the XADD
 * commands. */
typedef struct streamIterator {
    stream *stream;         /* The stream we are iterating. */
    streamID master_id;     /* ID of the master entry at listpack head. */
    uint64_t master_fields_count;       /* Master entries # of fields. */
    unsigned char *master_fields_start; /* Master entries start in listpack. */
    unsigned char *master_fields_ptr;   /* Master field to emit next. */
    int entry_flags;                    /* Flags of entry we are emitting. */
    int rev;                /* True if iterating end to start (reverse). */
    uint64_t start_key[2];  /* Start key as 128 bit big endian. */
    uint64_t end_key[2];    /* End key as 128 bit big endian. */
    raxIterator ri;         /* Rax iterator. */
    unsigned char *lp;      /* Current listpack. */
    unsigned char *lp_ele;  /* Current listpack cursor. */
    unsigned char *lp_flags; /* Current entry flags pointer. */
    /* Buffers used to hold the string of lpGet() when the element is
     * integer encoded, so that there is no string representation of the
     * element inside the listpack itself. */
    unsigned char field_buf[LP_INTBUF_SIZE];
    unsigned char value_buf[LP_INTBUF_SIZE];
} streamIterator;

listpack紧凑列表

        除了radix，stream中还有一种数据结构：紧凑列表，用于存储结点数据。从名字就可以看出，紧凑列表是一种结构非常紧凑的数据结构，由于篇幅原因这里不详细介绍了。感兴趣的同学可以研究下。

消费组功能实现

        消费组相关功能也是通过radix结构实现，其中streamCG结构保存消费组相关信息。

/* Consumer group. */
typedef struct streamCG {
    streamID last_id;       /* Last delivered (not acknowledged) ID for this
                               group. Consumers that will just ask for more
                               messages will served with IDs > than this. */
    rax *pel;               /* Pending entries list. This is a radix tree that
                               has every message delivered to consumers (without
                               the NOACK option) that was yet not acknowledged
                               as processed. The key of the radix tree is the
                               ID as a 64 bit big endian number, while the
                               associated value is a streamNACK structure.*/
    rax *consumers;         /* A radix tree representing the consumers by name
                               and their associated representation in the form
                               of streamConsumer structures. */
} streamCG;

/* A specific consumer in a consumer group.  */
typedef struct streamConsumer {
    mstime_t seen_time;         /* Last time this consumer was active. */
    sds name;                   /* Consumer name. This is how the consumer
                                   will be identified in the consumer group
                                   protocol. Case sensitive. */
    rax *pel;                   /* Consumer specific pending entries list: all
                                   the pending messages delivered to this
                                   consumer not yet acknowledged. Keys are
                                   big endian message IDs, while values are
                                   the same streamNACK structure referenced
                                   in the "pel" of the consumer group structure
                                   itself, so the value is shared. */
} streamConsumer;

/* Pending (yet not acknowledged) message in a consumer group. */
typedef struct streamNACK {
    mstime_t delivery_time;     /* Last time this message was delivered. */
    uint64_t delivery_count;    /* Number of times this message was delivered.*/
    streamConsumer *consumer;   /* The consumer this message was delivered to
                                   in the last delivery. */
} streamNACK;

总结

        截至目前为止，stream类型应该还没有在各大厂应用，Kafka的地位仍然稳固。当然多一种选择也没什么坏处，如果你有消息队列相关需求，但又觉得Kafka有点笨重，不妨试一下stream。即插即用，简单方便。