消息中间件 rabbitmq 级联-Shovel

Shovel plugin

Sometimes it is necessary to reliably and continually move messages from a queue (a source) in one broker to an exchange in another broker (a destination). The Shovel plugin allows you to configure a number of shovels, which do just that and start automatically when the broker starts.

The source queue and destination exchanges can be on the same broker (typically in different vhosts) or distinct brokers.

A shovel behaves like a well-written client application, which connects to its source and destination, reads and writes messages, and copes with connection failures. In fact, Shovels useRabbitMQ Erlang client under the hood.

The primary advantages of a shovel are:

Loose coupling

A shovel can move messages between brokers (or clusters) in different administrative domains:

• they may have different users and virtual hosts;
• they may run on different versions of RabbitMQ and Erlang.

WAN-friendly

The Shovel plugin uses AMQP 0-9-1 to communicate between brokers, and is designed to tolerate intermittent connectivity without message loss.

Highly tailorable

When a shovel connects (either to the source or the destination) it can be configured to perform any number of explicit methods. For example, the source queue need not exist initially, and can be  declared on connect.

A comparison between clustering, federated exchanges and shovels is given on the Distributed Messaging page.

What does it do?

The Shovel plugin defines (and runs) an Erlang client application for each shovel.

In essence, a shovel is a simple pump. Each shovel:

• connects to the source broker and the destination broker,
• consumes messages from the queue,
• re-publishes each message to the destination broker (using, by default, the original exchange name and routing_key).

The shovel configuration allows each of these processes to be tailored.

connects

After connection to a source or a destination broker a series of configured AMQP declarations can be issued. Queues, exchanges and bindings can be declared.

A shovel will attempt to reconnect to a broker if a failure occurs and multiple brokers can be specified for the source and destination so that another broker may be selected (at random) to reconnect to. A reconnection delay can be specified to avoid flooding the network with reconnection attempts, or to prevent reconnection on failure altogether.

All configured declarations (for that source or destination) are re-issued upon re-connect.

consumes

The shovel's consumer can acknowledge messages automatically on receipt, after (re-)publication, or after confirmation of its publication.

re-publishes

Both the publish method and the message properties can be modified with explicit parameter values.

Getting started

The Shovel plugin is included in the RabbitMQ distribution. To enable it, use rabbitmq-plugins:

rabbitmq-plugins enable rabbitmq_shovel

You may also wish to enable the rabbitmq_shovel_management plugin (see below).

There is no requirement to run the shovel on the same broker (or cluster) as its source or destination; the shovel can be entirely separate.

There are two distinct ways to define shovels: static shovels and dynamic shovels. The differences are as follows:

Static Shovels	Dynamic Shovels
Defined in the broker configuration file.	Defined in the broker's parameters.
Require a restart of the hosting broker to change.	Can be created and deleted at any time.
Slightly more general: any queues, exchanges or bindings can be declared manually at startup.	Slightly more opinionated: the queues, exchanges and bindings used by the shovel will be declared automatically.

Authentication and Authorization for Shovels

The Shovel plugin uses Erlang client under the hood to open connections to its source and/or destination. Just like any other client library connection, a Shovel connection must successfully authenticate with RabbitMQ server and be authorized to access the vhost it is trying to connect to.

Authentication and authorization failures of Shovel connections will be visible in the logs of the node that opens them.

Shovels between clusters

It's normally desirable to ensure that shovels are resilient to failure of any node in the source or destination clusters, or the cluster hosting the shovel.

You can ensure that shovels can fail over to different nodes in the source or destination clusters by specifying multiple source and/or destination URIs for each shovel.

Dynamic shovels are automatically defined on all nodes of the hosting cluster on which the shovel plugin is enabled. Each shovel will only start on one node (arbitrarily), but will fail over to another node if that one goes down.

Static shovels should be defined in the configuration file for all nodes of the hosting cluster on which the shovel plugin is enabled. Again each shovel will only start on one node and fail over when needed.

Securing Shovel Connections with TLS

Shovel connections can be secured with TLS. Because Shovel uses a RabbitMQ client under the hood, it is necessary to both configure source broker to listen for TLS connections and Shovel/Erlang client to use TLS.

To configure Shovel to use TLS, one needs to

• Specify CA certificate and client certificate/key pair via URI query parameters
• Configure Erlang client to use TLS

Just like with "regular" client connections, server's CA should be trusted on the node where Shovel runs, and vice versa.

Monitoring shovels

There are two ways of discovering the status of shovels.

Use shovel management

Shovel status can be reported on the Management plugin user interface by enabling the rabbitmq_shovel_management plugin wherever you have the management plugin enabled.

Information about configured shovels will automatically appear in the management API and UI.

Direct query

Shovel status can be obtained by direct query of the Shovel plugin app. Issue the following rabbitmqctl command:

  $ rabbitmqctl eval 'rabbit_shovel_status:status().'

This calls the status method in a module of the rabbitmq_shovel plugin, which will return an Erlang list, with one element for each configured shovel.

Each element of the list is a tuple with four fields: {Name,Type, Status,Timestamp}.

• Name is the shovel name,
• Type is either static or dynamic,
• Status is the current shovel state,
• and Timestamp is the time when the shovel entered this state.

  Timestamp is a local calendar time of the form {{YYYY, MM, DD}, {HH, MM, SS}}.

  Status takes one of three forms:

  • The shovel is starting up, connecting and creating resources:

    'starting'
    

  • The shovel is running normally:

    { 'running', [{'src_uri', Source},
                  {'dest_uri', Destination}]}:
    

    where Source and Destination terms give the respective connection parameters.

  • The shovel has terminated:

    {'terminated', Reason}
    

    where Reason is an Erlang term that indicates the reason for the termination.

  https://www.rabbitmq.com/shovel.html#management-status