Upgrading RabbitMQ Using Blue-Green Deployment Strategy

RabbitMQ

Features
Get Started
Support
Community
Docs
Blog

Overview

Blue-green deployment is an upgrade strategy that is based on the idea of to setting up a second RabbitMQ cluster (the “green” one) next to the current production cluster (the “blue” one). Applications are then switched to the “green” cluster. When that migration is done, the “blue” cluster is decomissioned (shut down). To simplify the switch, federated queues can be used to transfer enqueued messages from the “blue” to the “green” cluster.
Preparing the “green” Cluster

After deploying a brand new “green” cluster, there are two steps to follow:

import definitions, e.g. exchanges, queues, bindings;
configure federation to later drain messages.

Importing definitions

The procedure of definitions export/import is covered in the Backup guide. The “blue” is the source cluster and the “green” one is the target.
Configuring Queue Federation

RabbitMQ Federation plugin makes it easy to move consumers from “blue” to “green”, without disrupting message consumption or losing messages. The principle of federated queues is that the consumers now connected to “green” will get messages published to “blue” as long as there are no consumers in “blue” (local consumers take precedence).

Here is an example to federate all queues. In this example, the “blue” cluster is the upstream and the “green” one is the downstream.

First define the upstream on “green” and point it to “blue”:

rabbitmqctl set_parameter federation-upstream blue
‘{“uri”:“amqp://node-in-blue-cluster”}’

Then define a policy matching all queues which configure blue as the upstream:

rabbitmqctl set_policy --apply-to queues blue-green-migration “.*”
‘{“federation-upstream”:“blue”}’

Please read the guides linked above and the federation reference for further details.
Migrate Consumers Over

You can now switch your consumers to use the new “green” cluster. To achieve that, reconfigure your load balancer or your consumer applications, depending on your setup. The Upgrade guide covers some client features which enable them to switch between nodes.

At that point, your producers are still publishing to “blue”, but thanks to the federation plugin, message are transfered to consumers connected to “green”.
Drain Messages

The next step would be to switch producers to “green” as well. However, you may still have a backlog of messages in “blue”. The federation plugin doesn’t help here because it doesn’t move messages, it only allows remote consumers to dequeue messages.

In case of a large backlog, use the Shovel plugin on “green” to really drain messages in “blue”. This would require doing something like the following for each queue with a backlog:

rabbitmqctl set_parameter shovel drain-blue
‘{“src-protocol”: “amqp091”, “src-uri”: “amqp://node-in-blue-cluster”,
“src-queue”: “queue1”, “dest-protocol”: “amqp091”,
“dest-uri”: “amqp://”, “dest-queue”: “queue1”}’

Migrate Producers Over

Once the queues in “blue” are almost empty, you can stop producers. If message ordering is important to you, you should still wait a bit more so that the federation or shovel plugins finish to drain the queues on “blue”.

When they are empty, reconfigure your producers like you did for the consumers and start them again. At this point, everything is moved to the “green” cluster.
Decomission the “blue” Cluster

You are now free to shutdown the nodes in the “blue” cluster.
Real-world Example

Dan Baskette, Gareth Smith and Claude Devarenne of Pivotal published an article about this method where producers and consumers are CloudFoundry applications. The article is very detailed and uses diagrams to describe the procedure. They also made a video to show it in action.

This guide is inspired by their great work.
Getting Help and Providing Feedback

If you have questions about the contents of this guide or any other topic related to RabbitMQ, don’t hesitate to ask them on the RabbitMQ mailing list.
Help Us Improve the Docs ❤️

If you’d like to contribute an improvement to the site, its source is available on GitHub. Simply fork the repository and submit a pull request. Thank you!
In This Section

Install: Windows
Install: Debian and Ubuntu
Install: RPM-based Linux
Install: Homebrew
Install: Windows (manual)
Install: Generic binary build
Install: Solaris
Install: EC2
Upgrade
Blue-green deployment-based upgrade
Supported Platforms
Changelog
Erlang Versions
Signed Packages
Java Client Downloads
.NET Client Downloads
Erlang Client Downloads
Community Plugins
Snapshots

RabbitMQ

Features
Get Started
Support
Community
Docs
Blog

Copyright © 2007-2020 VMware, Inc. or its affiliates. All rights reserved. Terms of Use, Privacy and Trademark Guidelines