Posted by Bhavin Turakhia
We need a simple message queue to ensure asynchronous message passing across a bunch of our server side apps. The message volume is not intended to be very high, latency is not an issue, and order is not important, but we do need to guarantee that the message will be received and that there is no potential for failure irrespective of infrastructure downtime.
Dhruv from my team had taken up the task of researching various persistent message queue options and compiling notes on them. This is a compendium of his notes (disclaimer – this is an outline of our experience, there may be inaccuracies) -
RabbitMQ
General:
* Some reading on clustering http://www.rabbitmq.com/clustering.html
* DNS errors cause the DB(mnesia) to crash
* A RabbitMQ instance won’t scale to LOTS of queues with each queue having fair load since all queues are stored in memory (queue metadata) and also in a clustered setup, each queue’s metadata (but not the queue’’s messages) is replicated on each node. Hence, there is the same amount of overhead due to queues on every node in a cluster
* No ONCE-ONLY semanamntics. Messages may be sent twice by RabbitMQ to the consumer(s)
* Multiple consumers can be configured for a single queue, and they will all get mutually exclusive messages
* Unordered; not FIFO delivery
* Single socket multiple connections. Each socket can have multiple channels and each channel can have multiple consumers
* No provision for ETA
* maybe auto-requeue (based on timeout) — needs investigation
* Only closing connection NACKs a message. Removing the consumer from that channel does NOT. Hence, all queues being listened to on that channel/connetion are closed for the current consumer
* NO EXPONENTIAL BACKOFF for failed consumers. Failed messages are re-tried almost immediately. Hence an error in the consumer logic that crashes the consumer while consuming a particular message may potentially block the whole queue. Hence, the consumer needs to be programmed well — error free. However, apps are like; well apps…
We need a simple message queue to ensure asynchronous message passing across a bunch of our server side apps. The message volume is not intended to be very high, latency is not an issue, and order is not important, but we do need to guarantee that the message will be received and that there is no potential for failure irrespective of infrastructure downtime.
Dhruv from my team had taken up the task of researching various persistent message queue options and compiling notes on them. This is a compendium of his notes (disclaimer – this is an outline of our experience, there may be inaccuracies) -
RabbitMQ
General:
* Some reading on clustering http://www.rabbitmq.com/clustering.html
* DNS errors cause the DB(mnesia) to crash
* A RabbitMQ instance won’t scale to LOTS of queues with each queue having fair load since all queues are stored in memory (queue metadata) and also in a clustered setup, each queue’s metadata (but not the queue’’s messages) is replicated on each node. Hence, there is the same amount of overhead due to queues on every node in a cluster
* No ONCE-ONLY semanamntics. Messages may be sent twice by RabbitMQ to the consumer(s)
* Multiple consumers can be configured for a single queue, and they will all get mutually exclusive messages
* Unordered; not FIFO delivery
* Single socket multiple connections. Each socket can have multiple channels and each channel can have multiple consumers
* No provision for ETA
* maybe auto-requeue (based on timeout) — needs investigation
* Only closing connection NACKs a message. Removing the consumer from that channel does NOT. Hence, all queues being listened to on that channel/connetion are closed for the current consumer
* NO EXPONENTIAL BACKOFF for failed consumers. Failed messages are re-tried almost immediately. Hence an error in the consumer logic that crashes the consumer while consuming a particular message may potentially block the whole queue. Hence, the consumer needs to be programmed well — error free. However, apps are like; well apps…