QPID基础知识

一 AMQP

1.1 AMQP简介

AMQP（Advanced Message Queuing Protocol）是一种消息协议 ，等同于JMS，但是JMS只是java平台的方案，AMQP是一个跨语言的协议。AMQP 不分语言平台,最初来自JPMorgon,因为业务需要而滋生，很快从金融界推广到整个计算机界流行起来。

Publisher —>Exchange —> MessageQueue —>Consumer

1.2 AMQP基础概念

• Server：又称Broker。接受客户端连接，实现AMQP的服务的实体。
• Connection：连接。应用程序与Broker的网络连接。
• Channel：是进行消息读写的通道，相当是传统的session概念。每个Channel代表一个会话任务，一个Connection可以创建多个channel。
• Message：服务器和应用程序之间传送的数据，本质上就是一段数据，由Properties和Body组成。
• Exchange：Accepts messages from Producers and routes them to Queues if the message meets the criteria expressed in a binding.
  Exchange分为三种类型：
  • Fan-Out: Messages are routed to every Queue bound to the Exchange, ignoring the Routing Key.
  • Direct: A message is routed only if a Queue’s Binding Key is the same as the message’s Routing Key.
  • Topic: Similar to a Direct Exchange, but it supports multipart keys that contain multiple words separated by the “.” delimiter; for instance, a message Producer can create messages with Routing Keys like usa.news, usa.weather, europe.news, and europe.weather.
• Binding：Defines the relationship between an Exchange and a Queue, specifying which messages should be routed to a given Queue.
• Routing key：A string that the Exchange can use to determine to which Queues the message should be delivered.
• Queue：Holds messages and delivers them to the Consumers that subscribe to the Queue.
• Virtual Host：其实是一个虚拟概念，类似于权限控制组，一个Virtual Host里面可以有若干个Exchange和Queue，可以用来隔离Exchange和Queue，同一个Virtual Host 里面不能有相同名称的Exchange或Queue 。不同的Virtual Host 里面可以有相同名称的Exchange或Queue 。但是权限控制的最小粒度是Virtual Host。
• Producer: A program that writes messages to an Exchange. To do this, the program creates a message, fills the message with content, gives the message a Routing Key, and sends the message to an Exchange.
• Consumer：A program that reads messages from a Queue. A Consumer can create, subscribe to, share, use, or destroy Queue and their Bindings(as long as it has have permission to do so).

1.3 常见的AMQP协议实现

• RabbitMQ http://www.rabbitmq.com/

• Apache Qpid http://qpid.apache.org/

• OpenAMQ http://www.openamq.org/

二 QPID基础概念

QPID是AMQP的一个实现，提供C++，JAVA版本的broker，支持多种语言客户端。

2.1 Broker Federation

Broker Federation allows messaging networks to be defined by creating message routes, in which messages in one broker (the source broker) are automatically routed to another broker (the destination broker). These routes may be defined between exchanges in the two brokers (the source exchange and the destination exchange), or from a queue in the source broker (the source queue) to an exchange in the destination broker.
Message routes are unidirectional; when bidirectional flow is needed, one route is created in each direction.
Routes can be durable or transient. A durable route survives broker restarts, restoring a route as soon as both the source broker and the destination are available. If the connection to a destination is lost, messages associated with a durable route continue to accumulate on the source, so they can be retrieved when the connection is reestablished.

The destination for a route is always an exchange on the destination broker. By default, a message route is created by configuring the destination broker, which then contacts the source broker to subscribe to the source queue. This is called a pull route. It is also possible to create a route by configuring the source broker, which then contacts the destination broker in order to send messages. This is called a push route, and is particularly useful when the destination broker may not be available at the time the messaging route is configured, or when a large number of routes are created with the same destination exchange.

The source for a route can be either an exchange or a queue on the source broker. If a route is between two exchanges, the routing criteria can be given explicitly, or the bindings of the destination exchange can be used to determine the routing criteria. To support this functionality, there are three kinds of message routes: queue routes, exchange routes, and dynamic exchange routes.

• Queue Routes route all messages from a source queue to a destination exchange. If message acknowledgement is enabled, messages are removed from the queue when they have been received by the destination exchange; if message acknowledgement is off, messages are removed from the queue when sent.

• Exchange routes route messages from a source exchange to a destination exchange, using a binding key (which is optional for a fanout exchange). Internally, creating an exchange route creates a private queue (auto-delete, exclusive) on the source broker to hold messages that are to be routed to the destination broker, binds this private queue to the source broker exchange, and subscribes the destination broker to the queue.

• Dynamic exchange routes allow a client to create bindings to an exchange on one broker, and receive messages that satisfy the conditions of these bindings not only from the exchange to which the client created the binding, but also from other exchanges that are connected to it using dynamic exchange routes. If the client modifies the bindings for a given exchange, they are also modified for dynamic exchange routes associated with that exchange.

参考：qpid官网 Home->Releases->Qpid C++ 1.39.0->AMQP Messaging Broker (Implemented in C++)->1.4. Broker Federation

三 QPID使用

3.1 QPID 配置案例

3.1.1 producer -> broker1(Exchange) -> broker2(Exchange) -> consumer

• 在broker1上add exchange

qpid-config add exchange topic MarketData-EX --durable

• 在broker2上add exchange

qpid-config add exchange topic MarketData-EX  --durable

• 在broker2上增加broker1到broker2之间的routing

qpid-route --ack 100 --duralbe route add broker2_host:broker2_port broker1_host:broker1_port MarketData-EX  MD.Stock.#
qpid-route --ack 100 --duralbe route add broker2_host:broker2_port broker1_host:broker1_port MarketData-EX  MD.Future.#

• 在broker2上检查新增的routing

qpid-route route list
qpid-route link list

3.1.2 producer -> broker1(Exchange) -> broker2(Exchange->LVQ) -> consumer

• 在broker1上add exchange

qpid-config add exchange topic MarketData-EX --durable

• 在broker2上add exchange

qpid-config add exchange topic MarketData-EX  --durable

• 在broker2上add LVQ 并且bind 到 exchange

qpid-config add queue LVQ_MarketData --lvq-key=Symbol --durable 
qpid-config bind MarketData-EX LVQ_MarketData MD.Stock.#  --durable 

• 在broker2上增加broker1到broker2之间的routing

qpid-route --ack 100 --dualbe route add broker2_host:broker2_port broker1_host:broker1_port MarketData-EX  MD.#.#

• 在broker2上检查新增的routing

qpid-route route list
qpid-route link list
qpid-config list binding #列出所有binding

通过（Exhange->LVQ）方式订阅的， 在qpid-stat -u 显示订阅时，显示的不是LVQ而是Exchange名字。

3.2 QPID 常用命令

3.2.1 qpid-config

用来完成诸如添加 / 删除 Queue，添加 / 删除 Exchange 等关于 broker 内部的配置工作。

• 列出Exchanges和Queues的统计
  

• qpid-config queues 列出queues详细信息
  

• qpid-config exchanges 列出exchanges详细信息
  

3.2.2 qpid-route

用来配置 broker Federation。

3.2.3 qpid-tool

用来对 Qpid Broker 进行实时监控。

3.2.4 qpid-stat

用来查看qpid的统计情况。
-g General统计
-c connection统计
-e exchange统计
-q queue统计
-u 订阅统计
-m 内存统计

3.3 QPID 高可用

3.4 QPID 发布端

3.5 QPID 接收端

3.6 QPID 性能和延迟

每条消息的大小会影响接收性能； 接收端ack和capacity的大小设置也会影响接收性能。

• Receiver Capacity (Prefetch)
  By default, a receiver requests the next message from the server in response to each fetch call, resulting in messages being sent to the receiver one at a time. As in the case of sending, it is often desirable to avoid this roundtrip for each message. This can be achieved by allowing the receiver to prefetch messages in anticipation of fetch calls being made. The receiver needs to be able to store these prefetched messages, the number it can hold is controlled by the receivers capacity.

• Acknowledging Received Messages
  Applications that receive messages should acknowledge their receipt by calling the session’s acknowledge method. As in the case of sending messages, acknowledged transfer of messages to receivers provides at-least-once reliability, which means that the loss of the connection or a client crash does not result in lost messages; durable messages are not lost even if the broker is restarted. Some cases may not require this however and the reliability can be controlled through a link property in the address options (see Table 1.3, “Link Properties”). The acknowledge call acknowledges all messages received on the session (i.e. all message that have been returned from a fetch call on a receiver created on that session). The acknowledge call also support an optional parameter controlling whether the call is synchronous or not.
  【参考qpid官方文档】

3.6.1 Performance Tips

• Consider prefetching messages for receivers (see Section 1.6, “Receiver Capacity (Prefetch)”). This helps eliminate roundtrips and increases throughput. Prefetch is disabled by default, and enabling it is the most effective means of improving throughput of received messages.

• Send messages asynchronously. Again, this helps eliminate roundtrips and increases throughput. The C++ and .NET clients send asynchronously by default, however the python client defaults to synchronous sends.

• Acknowledge messages in batches (see Section 1.7, “Acknowledging Received Messages”). Rather than acknowledging each message individually, consider issuing acknowledgements after n messages and/or after a particular duration has elapsed.

• Tune the sender capacity (see Section 1.5, “Sender Capacity and Replay”). If the capacity is too low the sender may block waiting for the broker to confirm receipt of messages, before it can free up more capacity.

• If you are setting a reply-to address on messages being sent by the c++ client, make sure the address type is set to either queue or topic as appropriate. This avoids the client having to determine which type of node is being refered to, which is required when hanling reply-to in AMQP 0-10.

• For latency sensitive applications, setting tcp-nodelay on qpidd and on client connections can help reduce the latency.
  【参考资料】

3.7 消息丢失和重发机制

3.8 QPID 常见问题

3.8.1 案例1

• 问题描述：qpid日志报错，Maximum depth exceeded, 并且运行qpid-stat -g显示queue-depth大于100万（即消息堆积数量超过100万）
  
  
• 问题查看： 运行qpid-stat -u 命令，并grep 日志里面看到的queue名字，就能找到发生问题的订阅的具体信息（包括对端ip、端口号，甚至进程名字和processID）
  
• 问题原因：通过查看procName和procId找到了出问题的接收端， 查看接收端收到的消息，发现收到的消息里面有非常多的重复消息，并且重复了很多次。 排查了发送端没有问题， 想到可能是因为route路径上面有消息回环导致的。 经过检查，发现当日新增route时有错误，dest-broker 跟 src-broker 位置写反了， 跟原来已有的route形成了回环，消息在两个broker之间不断的传输，导致接收端收到大量重复消息。

3.8.2 案例2

• 问题描述：有两个broker，broker1和 broker2 ， 在 broker2加了一个route， 从broker1 到 broker2传输消息。broker1重启之后， broker1和broker2的日志有错误，并且 route上无法继续传输消息 。
  broker1日志:
  
  broker2日志:
  
  此时在broker1 使用qpid-stat -c 查看 connection， 可以看到connection是正常的：
  
  此时在broker2使用qpid-stat -c 查看connection， 或者使用qpid-route link list 查看route， 也都是正常的：
  
  
  要想发现问题，需要在broker1上面去查看queue情况， 因为每添加一个route， 会在source broker （在这里也就是borker1）上面新增一个queue， 如果查看queue没有对应的queue，就说明borker2重启后router没有连接成功。
  从qpid-stat 找到新建立的连接的端口号11085， 然后运行qpid-stat -u | grep 11085, 如果有对应router的queue建立成功，就说明route连接好了。 如果没有queue建立，则说明route没有连接好。

• 临时解决方案： 将broker2的route删了重新加，可以解决问题。

• 原因分析：broker2重启，但是broker1并不知道，还保留着之前的session。 broker2重启后尝试连接broker1， 但是broker1因为有之前的session，所以拒绝了新的连接。

• 解决方案：broker1或者2启动的时候，使用–link-heartbeat-interval 5 选项，从应用层发hearbeat，如果发现连接有问题了就会重建连接。 第二种解决方案是在重启broker2时先等待一段时间，比如2分钟。
  参考：https://access.redhat.com/solutions/56487

Resolution
Two workarounds exists (applicable to MRG Messaging or standalone qpid installation):

Use heartbeats to check connection 'liveness' on application layer. That is, use option heartbeat:5 when defining the session. Then every 5 seconds, a connection probe will be sent from the client. If two successive probes are not responded, the session is considered to be lost and TCP connection is dropped properly (and due to reconnect:true the session will be reestablished). The 5 seconds can be modified per maximal TCP retransmission time (the time elapsed between the first and last retransmission of the non-ACK-ed packet). To let the heartbeats effective, use any value lower than 1/2 of the maximal retransmission time.

Please note that changing connection options reconnect-timeout, reconnect-limit, reconnect-interval-min and reconnect-interval-max might improve the behaviour even more. Setting first reconnect timeout to more than 2*heartbeat will cause the broker already detected timed out connection and dropped it, so no "Session already attached" error will occur at all.
Note also that increasing those connection options can prevent the case that outage takes longer than all retry attempts and the client gives up to reconnect completely.
Align number of TCP retransmissions so that any (expected) network outage is shorter than the full TCP retransmission round. Then, some  retransmission will be successfull and no disconnection would happen.

For issues with Satellite 6, please contact Red Hat support prior to applying the workarounds.

Root Cause
Simulation of network outage (i.e. iptables blocking) took longer than TCP retransmissions so OS gave up to retransmit the non-ACK-ed packet.
When TCP retransmission gives up, kernel sends ETIMEDOUT to userspace and moves the local socket from ESTABLISHED to CLOSED status - i.e. no RST packet is sent on the connection. This results in situation where:

the client does not have the TCP connection established

the server has the connection established (in ESTABLISHED state even)

the server has not received any indication that the client has some transmitting problems.

The above apparently follows relevant RFCs.
The server can not detect the TCP connection being broken only when sending TCP keepalive probe, because qpid sockets does not implements SO_KEEPALIVE. Meanwhile, the server still sees the old connection established.
When client tries to reconnect to the server using new TCP connection, it uses the same connection ID (as the client tries to reconnect to previously established connection only). Broker rejects the incoming session request as it is duplicate (from its point of view).

FAQ

• 1 接收端从Exchange接收数据，与从Queue接收数据，有什么不同？
  • 从接收效果看：
    从Exchange接收数据，当前连上来的所有接收端都能收到一个数据。 如果一个接收端是后面连上来的，就收不到前面的消息了。
    从Queue接收数据， 每个数据只会被一个接收端取走，其他接收端就接收不到了。 如果当前没有接收端，数据会被存放在Queue里面，等到有接收端连上了可以收到前面的数据。 （接收端从Queue接收数据的时候可以设置一个BrowseOnly的选项，这样该接收端就不会把数据取走）
  • 从接收Address看：
    Qpid Address 表示一个节点，有两种节点：一种是 queue，另外一种是 topic（也就是上面所说的从Exchange接收）。Queue 节点能够缓存消息，直到被读取走为止；而 topic 节点则即时进行转发，比如假如有 4 个 consumer 对某消息感兴趣，当消息到达节点时，有 3 个 consumer 正在运行，那么 topic 节点会将消息转发给这 3 个 consumer，然后就将该消息丢弃。剩下的那个 consumer 再运行时，则收不到这个消息。

参考资料

$ export QPID_LOG_ENABLE=“warning+”
export QPID_LOG_TO_FILE=“/tmp/myclient.out”
【1】Programming in Apache Qpid http://people.apache.org/~jonathan/Programming-In-Apache-Qpid.html
【2】https://blog.csdn.net/hzrandd/article/details/10239651
【1】https://www.amqp.org/ AMQP官网
【2】https://www.jianshu.com/p/c6eab4d6ff5c AMQP协议介绍

​