Python 全栈系列193 Redis Stream再探:官方教程研读笔记

说明

之前基本是靠“猜猜看”的方式来使用redis stream的，然后有些地方就很困惑，甚至毁三观。我发现其实没有很好的中文资料说明,最后就只好去看了官方的文档。我发现其实说的还挺清晰的，开卷还是有益。

另外我觉得python的redis包不是很好用(pip install redis), 问题可能还是在于redis本身的版本迭代相对快，所以版本很难完美适配。不像pymongo,基本上我就直接用包提供的方法就好。

所以我就把redis定位成py2neo一样，主要提供连接以及命令的透传，自己来提供命令模板。我觉得对于一些较新的数据库都可以用这种思路，尽量用数据库提供的原生语言，毕竟我们安装的包总是落后数据库本身很多。

内容

以下研读的内容来自官方的教程 Redis Streams tutorial，我可能会稍微打乱一下内容的顺序。

1 延时测试

我觉得可以先说延时方面的结果。官方用比较老的机器做了延时测试，并且每次操作一万条消息（这倒是比较符合我的应用），所以实际使用延时会更小一些。在测试中，99%的消息都在2ms内处理完毕，并且说多几百万条消息也是差不多，这点挺好的，对我来说够用了。

---

In order to check these latency characteristics a test was performed using multiple instances of Ruby programs pushing messages having as an additional field the computer millisecond time, and Ruby programs reading the messages from the consumer group and processing them. The message processing step consisted of comparing the current computer time with the message timestamp, in order to understand the total latency.

Processed between 0 and 1 ms -> 74.11%
Processed between 1 and 2 ms -> 25.80%
Processed between 2 and 3 ms -> 0.06%
Processed between 3 and 4 ms -> 0.01%
Processed between 4 and 5 ms -> 0.02%

So 99.9% of requests have a latency <= 2 milliseconds, with the outliers that remain still very close to the average.

Adding a few million unacknowledged messages to the stream does not change the gist of the benchmark, with most queries still processed with very short latency.

A few remarks:

• Here we processed up to 10k messages per iteration, this means that the COUNT parameter of XREADGROUP was set to 10000. This adds a lot of latency but is needed in order to allow the slow consumers to be able to keep with the message flow. So you can expect a real world latency that is a lot smaller.
• The system used for this benchmark is very slow compared to today’s standards.

---

2 与kafka的比较

redis队列的设计有借鉴kafka的地方，挑重点说，redis的队列相当于实现了服务端的负载均衡。worker能力强就会多分配。而kafka更像是n条redis 队列，这种分片拓展我想redis是认为用户可以自己搞。

言下之意就是，redis提供的这种机制更棒棒，至于能力的横向拓展你自己来。这种模式的确也更符合我的要求，我希望能够快速缓存/处理用户的需求。

---

Differences with Kafka ™ partitions
Consumer groups in Redis streams may resemble in some way Kafka ™ partitioning-based consumer groups, however note that Redis streams are, in practical terms, very different. The partitions are only logical and the messages are just put into a single Redis key, so the way the different clients are served is based on who is ready to process new messages, and not from which partition clients are reading. For instance, if the consumer C3 at some point fails permanently, Redis will continue to serve C1 and C2 all the new messages arriving, as if now there are only two logical partitions.

Similarly, if a given consumer is much faster at processing messages than the other consumers, this consumer will receive proportionally more messages in the same unit of time. This is possible since Redis tracks all the unacknowledged messages explicitly, and remembers who received which message and the ID of the first message never delivered to any consumer.

However, this also means that in Redis if you really want to partition messages in the same stream into multiple Redis instances, you have to use multiple keys and some sharding system such as Redis Cluster or some other application-specific sharding system. A single Redis stream is not automatically partitioned to multiple instances.

We could say that schematically the following is true:

• If you use 1 stream -> 1 consumer, you are processing messages in order.
• If you use N streams with N consumers, so that only a given consumer hits a subset of the N streams, you can scale the above model of 1 stream -> 1 consumer.
• If you use 1 stream -> N consumers, you are load balancing to N consumers, however in that case, messages about the same logical item may be consumed out of order, because a given consumer may process message 3 faster than another consumer is processing message 4.
  So basically Kafka partitions are more similar to using N different Redis keys, while Redis consumer groups are a server-side load balancing system of messages from a given stream to N different consumers.

---

3 五种符号在redis stream中的特殊意义

目前主要是-, +, $, > , * 这五种。

• 1 -,+ 表示了id的最小和最大值
• 2 $ 表示当前队列最后一个id，肯定是小于+的
• 3 > 获取未分配给其他消费者的消息
• 4 * 表示使用系统自动id

---

You may have noticed that there are several special IDs that can be used in the Redis API. Here is a short recap, so that they can make more sense in the future.

The first two special IDs are - and +, and are used in range queries with the XRANGE command. Those two IDs respectively mean the smallest ID possible (that is basically 0-1) and the greatest ID possible (that is 18446744073709551615-18446744073709551615). As you can see it is a lot cleaner to write - and + instead of those numbers.

Then there are APIs where we want to say, the ID of the item with the greatest ID inside the stream. This is what $ means. So for instance if I want only new entries with XREADGROUP I use this ID to signify I already have all the existing entries, but not the new ones that will be inserted in the future. Similarly when I create or set the ID of a consumer group, I can set the last delivered item to $ in order to just deliver new entries to the consumers in the group.

As you can see $ does not mean +, they are two different things, as + is the greatest ID possible in every possible stream, while $ is the greatest ID in a given stream containing given entries. Moreover APIs will usually only understand + or $, yet it was useful to avoid loading a given symbol with multiple meanings.

Another special ID is >, that is a special meaning only related to consumer groups and only when the XREADGROUP command is used. This special ID means that we want only entries that were never delivered to other consumers so far. So basically the > ID is the last delivered ID of a consumer group.

Finally the special ID *, that can be used only with the XADD command, means to auto select an ID for us for the new entry.

So we have -, +, $, > and *, and all have a different meaning, and most of the time, can be used in different contexts.

---

其实还有(这个符号，用在id前面，表示不包含词条消息，相当于大于号。默认情况redis的逻辑是大于等于。但是这个符号似乎在redis 7.x才开始用。所以这就是我说的，redis的有些功能随版本还是在不断变化的。甚至在命令层面，有个XAUTOCLAIM命令，可以批量的重新认领消息，但是在6.2才有，我是6.0的版本。Redis Stream本身也是从5.x才开始的。

下面这句的意思是把mystream里所有的超过3600秒的消息转交给Alice消费者，限制只转交1条。

XAUTOCLAIM mystream mygroup Alice 3600000 0-0 COUNT 1

4 一些理念/规范

最初我以为我猜对了，后来发现不是这样就有点毁三观，再后来发现其实和我最初猜的差不多

redis的操作很有linux的风格，很多命令很简洁，很多设计的思路更为巧妙。所以偏差在于这些地方，然后又因为简洁的命令风格，错了也就不能用。

首先是一个 append-only的数据结构，当然可以ack和del。

毫秒级时间戳 ID

<millisecondsTime>-<sequenceNumber>

redis使用毫秒级时间戳，再加上序号来给所有的消息编号。我之前在启动容器的多个分身时也是采用毫秒级时间戳，所以比较能理解这么设计的原因。

• 1 确实能区分不同的对象。当然redis对应的并发更大，所以又加了一层序号。
• 2 能够提供重要的时间信息。这个无论是分析还是应用都很有用，例如我只想回看过去10秒的信息，甚至于历史上某10秒的信息。

原文里也提到：

We could also see a stream in quite a different way: not as a messaging system, but as a time series store.
不仅是消息序列，也是时间序列

未分配消息

提供服务端的子集分发(subset)

这个功能其实很棒，这样我就不用再搞一个服务作为扎口，redis stream里本身就自带了这样的功能。具体来说就是使用 > 来启用服务的功能。

特别适合启动多个worker并行去处理消息。

However in certain problems what we want to do is not to provide the same stream of messages to many clients, but to provide a different subset of messages from the same stream to many clients.

消费组的目的是把一个流分成不同子集，交给消费者。这类问题就是并行处理问题，我们当然不希望worker/consumer反复的处理同样的数据。

反过来看，另一种大模式，或者是更主流的模式的确是把相同的消息给到多个消费者，有点类似视频流直播。可能对数据处理来说，我可以让多个不同版本的程序接受这种广播流。

两种获取消息的模式

一种就是无视一切，直接读取。例如xrange和xread。

还有一种就是消费者模式，xreadgroup

假定worker/consumer会失效

所以还特别搞了xclaim，我觉得对于失效的处理还有别的办法。一般我的worker会处理完，然后ack并del。所以只要去回看一段时间之前，或者之内的残余消息，拿出来再重新发布就可以了。

所以大概一脉相承的，worker去请求消息时要"自己说自己是谁"，服务端并不需要提前注册某个worker。文中也提到几次，这种消费组的分配仅仅是逻辑层面上的。所以在消费组模式下请求消息时，要声明自己是谁，我目前用的redis包就是在这个功能上让我confuse了。这个模式也是我目前分布式worker的主要模式，进一步可以加一层鉴权，避免“假冒” 。

A consumer group is like a pseudo consumer that gets data from a stream, and actually serves multiple consumers, providing certain guarantees:

Each message is served to a different consumer so that it is not possible that the same message will be delivered to multiple consumers.

Consumers are identified, within a consumer group, by a name, which is a case-sensitive string that the clients implementing consumers must choose. This means that even after a disconnect, the stream consumer group retains all the state, since the client will claim again to be the same consumer. However, this also means that it is up to the client to provide a unique identifier.

Each consumer group has the concept of the first ID never consumed so that, when a consumer asks for new messages, it can provide just messages that were not previously delivered.

Consuming a message, however, requires an explicit acknowledgment using a specific command. Redis interprets the acknowledgment as: this message was correctly processed so it can be evicted from the consumer group.

A consumer group tracks all the messages that are currently pending, that is, messages that were delivered to some consumer of the consumer group, but are yet to be acknowledged as processed. Thanks to this feature, when accessing the message history of a stream, each consumer will only see messages that were delivered to it.

消费组是有时间概念的

例如对已存在的队列创建消费组时就很明显。

5 redis操作对象的改造

有一些成熟的应用，我就仍然沿用redis包提供的函数，关于stream的操作我全部改为用命令执行。

5