pulsar-topic

概述

topics in Pulsar are named channels for transmitting messages from producers to consumers。

linux 中管道，1. 使用管道符通过 “｜”，左侧为写入端口，右侧为读取端 ；2. 单向管道，比如父子进程可以通过创建匿名管道进行通信。（ pipe 系统调用函数的参数列表中有一个数组，是一个输出型参数。）

• 自动创建机制：Pulsar creates that topic under the namespace provided in the topic name automatically. If no tenant or namespace is specified when a client creates a topic, the topic is created in the default tenant and namespace.

命名空间

• Namespaces ：topic 的集合，作用域在 tenant 内。namespace是Pulsar中最基本的管理单元，在namespace这一层面，可以设置权限，调整副本设置，管理跨集群的消息复制，控制消息策略和执行关键操作。
• 租户概念：a tenant with different applications can create a separate namespace for each application.
• topic ：一个名称空间下，可以定义多个Topic 通过Topic进行数据的分类划分，将不同的类别的消息放置到不同Topic，消费者也可以从不同Topic中获取到相关的消息，是一种更细粒度的消息划分操作

Subscriptions

• Subscriptions ： a named configuration rule that determines how messages are delivered to consumers.数据分发配置（负载策略），通过 subscribe 来进行一种负载策略和消费者的绑定关系。

Subscriptions type

• Exclusive： 默认配置， If multiple consumers subscribe to a topic using the same subscription, an error occurs.
• Failover：单词本意是故障转移，该模式下，为了确保消息能够正常被消费出去，通过设置多个 consumer 的思想来保证 consumer 的可用性，从而维持消息的稳定消费。When the master consumer disconnects, all (non-acknowledged and subsequent) messages are delivered to the next consumer in line.
  • 分片情况下：failover 机制下，每个分片会找一个 master consumer。
    • 分片大于consumer ，一个consumer 会被分配多个（轮询策略）
    • 分片数小于 consumer ，每个consumer 对应一个
  • 非分片情况下：
    • topic 数大于 consumer ，一个consumer 可负责多个topic
    • topic 数小于 consumer ，1 v 1 关系
• Shared ：allows multiple consumers to attach to the same subscription.
  

命名空间example：
A namespace allows the application to create and manage a hierarchy of topics. The topic my-tenant/app1 is a namespace for the application app1 for my-tenant.

key_shared 模式的key 分配规则

• Auto-split Hash Range

  • 默认 65536 进行取模
  • 具体算法：murmur32(“Order-3459134”) = 3112179635, and its index in the range would be 3112179635 mod 65536 = 6067.

• Auto-split Consistent Hashing ： mark 100 points on that circle and associate them to the newly added consumer.

• Sticky ： Every newly added consumer specifies the ranges it wishes to be mapped to by using Consumer API. When the consumer object is constructed, you can specify the list of ranges. It’s your responsibility to make sure there are no overlaps and all the range is covered by regions.

C1 = [0, 16384), [32768, 49152)
C2 = [16384, 32768), [49152, 65536)

 0               16,384            32,768           49,152             65,536
 |------- C1 ------|------- C2 ------|------- C1 ------|------- C2 ------|

key 重新负载：

Key Shared Subscription type guarantees a key will be processed by a single consumer at any given time. **When a new consumer is connected, some keys will change their mapping from existing consumers to the new consumer.**

When the consumers are using the Key_Shared subscription type, you need to disable batching or use key-based batching for the producers.

Subscription modes

The subscription mode indicates the cursor belongs to durable type or non-durable type. A durable subscription and a non-durable subscription can have the same name, 主要是维护消费位置（cursor）在broker重新链接后的策略。

• When a subscription is created, an associated cursor is created to record the last consumed position.

• When a consumer of the subscription restarts, it can continue consuming from the last message it consumes.

使用方式

        Consumer<byte[]> consumer = pulsarClient.newConsumer()
                .topic("my-topic")
                .subscriptionName("my-sub")
                .subscriptionMode(SubscriptionMode.Durable)
                .subscribe();

消息删除策略

https://pulsar.apache.org/docs/3.3.x/cookbooks-retention-expiry/

• 自动删除：被全部消费完成的，可以进行删除
• 没有指定 durable subscriptions ：messages of a topic without any durable subscriptions are marked as deleted. If you want to prevent the messages from being marked as deleted, you can create a durable subscription for this topic. In this case, only acknowledged messages are marked as deleted.