Message System

 

Database Table Queue vs Message System

http://stackoverflow.com/questions/255794/queues-against-tables-in-messaging-systems

If you are building a single system which is already using a database, you don't have very high performance throughput requirements and you don't have to communicate with any other teams or systems then you're probably right.

For simple, low thoughput, mostly single threaded stuff, database are a totally fine alternative to message queues.

Where a message queue shines is when

• you want a high performance, highly concurrent and scalable load balancer so you can process tens of thousands of messages per second concurrently across many servers/processes (using a database table you'd be lucky to process a few hundred a second and processing with multiple threads is pretty hard as one process will tend to lock the message queue table)
• you need to communicate between different systems using different databases (so don't have to hand out write access to your systems database to other folks in different teams etc)

For simple systems with a single database, team and fairly modest performance requirements - sure use a database. Use the right tool for the job etc.

However where message queues shine is in large organisations where there are lots of systems that need to communicate with each other (and so you don't want a business database to be a central point of failure or place of version hell) or when you have high performance requirements.

In terms of performance a message queue will always beat a database table - as message queues are specifically designed for the job and don't rely on pessimistic table locks (which are required for a database implementation of a queue - to do the load balancing) and good message queues will perform eager loading of messages to queues to avoid the network overhead of a database .

Similarly - you'd never use a database to do load balancing of HTTP requests across your web servers - as it'd be too slow - if you have high performance requirements for your load balancer you'd not use a database either.

 

 

Message System Sample - RabbitMQ

 

Getting started : http://www.rabbitmq.com/getstarted.html

 

Sample 1: Hello World

 

Sending:

 

//The connection abstracts the socket connection, and takes care of protocol version negotiation and authentication and so on for us. Next we create a channel, which is where most of the API for getting things done resides.
    ConnectionFactory factory = new ConnectionFactory();
    factory.setHost("localhost");
    Connection connection = factory.newConnection();
    Channel channel = connection.createChannel();
   
    //To send, we must declare a queue for us to send to; then we can publish a message to the queue:
    channel.queueDeclare(QUEUE_NAME, false, false, false, null);//Declaring a queue is idempotent; it will be created if it doesn't exist already. The message contents is a byte array, so you can encode whatever you like there.
   
    String message = "Hello World!";
    channel.basicPublish("", QUEUE_NAME, null, message.getBytes());
    System.out.println(" [x] Sent '" + message + "'");
   
    //Lastly, we close the channel and the connection;
    channel.close();
    connection.close();

Receiving:

 

ConnectionFactory factory = new ConnectionFactory();
    factory.setHost("localhost");
    Connection connection = factory.newConnection();
    Channel channel = connection.createChannel();

    channel.queueDeclare(QUEUE_NAME, false, false, false, null);
    System.out.println(" [*] Waiting for messages. To exit press CTRL+C");
   
    //We're about to tell the server to deliver us the messages from the queue. Since it will push us messages asynchronously, we provide a callback in the form of an object that will buffer the messages until we're ready to use them.
    QueueingConsumer consumer = new QueueingConsumer(channel);
    channel.basicConsume(QUEUE_NAME, true, consumer);

    while (true) {
      QueueingConsumer.Delivery delivery = consumer.nextDelivery();
      String message = new String(delivery.getBody());
      System.out.println(" [x] Received '" + message + "'");
    }
    //QueueingConsumer.nextDelivery() blocks until another message has been delivered from the server.

 

 

Sample 2: Work queues

 

In the first tutorial we wrote programs to send and receive messages from a named queue. In this one we'll create a Work Queue that will be used to distribute time-consuming tasks among multiple workers.

Round-robin dispatching
One of the advantages of using Task Queue is the ability to easily parallelise work. If we are building up a backlog of work, we can just add more workers and that way, scale easily.

 

Run two Receiving scripts.

 

 

Let's see what is delivered to our workers:

shell1$ 
java -cp .:commons-io-1.2.jar:commons-cli-1.1.jar:rabbitmq-client.jar
Worker
 [
*]
 Waiting for 
messages. To exit 
press CTRL+C
 [
x]
 Received 'First message.'

 [
x]
 Received 'Third message...'

 [
x]
 Received 'Fifth message.....'

java -cp .:commons-io-1.2.jar:commons-cli-1.1.jar:rabbitmq-client.jar
Worker
 [
*]
 Waiting for 
messages. To exit 
press CTRL+C
 [
x]
 Received 'Second message..'

 [
x]
 Received 'Fourth message....'

By default, RabbitMQ will send each message to the next consumer, in sequence. On average every consumer will get the same number of messages. This way of distributing messages is called round-robin

 

**Message acknowledgment

 

In order to make sure a message is never lost, RabbitMQ supports message  acknowledgments . An ack(nowledgement) is sent back from the consumer to tell RabbitMQ that a particular message has been received, processed and that RabbitMQ is free to delete it.

If a consumer dies without sending an ack, RabbitMQ will understand that a message wasn't processed fully and will redeliver it to another consumer. That way you can be sure that no message is lost, even if the workers occasionally die.

**There aren't any message timeouts ; RabbitMQ will redeliver the message only when the worker connection dies. It's fine even if processing a message takes a very, very long time.

 

 

QueueingConsumer
 consumer
 =
 new
 QueueingConsumer
(
channel
);

boolean
 autoAck
 =
 false
;

channel
.
basicConsume
(
"hello"
,
 autoAck
,
 consumer
);


while
 (
true
)
 {

  QueueingConsumer
.
Delivery
 delivery
 =
 consumer
.
nextDelivery
();

  //...      

  channel
.
basicAck
(
delivery
.
getEnvelope
().
getDeliveryTag
(),
 false
);

}

 

**Message durability

We have learned how to make sure that even if the consumer dies, the task isn't lost. But our tasks will still be lost if RabbitMQ server stops.

 

Two things are required to make sure that messages aren't lost: we need to mark both the queue and messages as durable.

 

First, we need to make sure that RabbitMQ will never lose our queue. In order to do so, we need to declare it as  durable :

boolean
 durable
 =
 true
;

channel
.
queueDeclare
(
"hello"
,
 durable
,
 false
,
 false
,
 null
);

 

 

At this point we're sure that the  queue won't be lost even if RabbitMQ restarts. Now we need to mark our messages as persistent - by setting  MessageProperties   (which implements  BasicProperties ) to the value  PERSISTENT_TEXT_PLAIN .

 

import
 com.rabbitmq.client.MessageProperties
;


channel
.
basicPublish
(
""
,
 "task_queue"
,
 
            MessageProperties
.
PERSISTENT_TEXT_PLAIN
,

            message
.
getBytes
());

 

**Fair dispatch

 

You might have noticed that the dispatching still doesn't work exactly as we want. For example in a situation with two workers, when all odd messages are heavy and even messages are light, one worker will be constantly busy and the other one will do hardly any work.

 

In order to defeat that we can use the  basicQos   method with the  prefetchCount   =  1 setting. This tells RabbitMQ not to give more than one message to a worker at a time. Or, in other words, don't dispatch a new message to a worker until it has processed and acknowledged the previous one. Instead, it will dispatch it to the next worker that is not still busy.

int
 prefetchCount
 =
 1
;

channel
.
basicQos
(
prefetchCount
);

 

Sample 3: Publish/Subscribe

 

In the  previous tutorial   we created a work queue. The assumption behind a work queue is that each task is delivered to exactly one worker. In this part we'll do something completely different -- we'll deliver a message to multiple consumers. This pattern is known as "publish/subscribe".

**Exchange

The core idea in the messaging model in RabbitMQ is that the producer never sends any messages directly to a queue. Actually, quite often the producer doesn't even know if a message will be delivered to any queue at all.

Instead, the producer can only send messages to an  exchange .

There are a few exchange types available:  direct ,  topic ,  headers   and  fanout . We'll focus on the last one -- the fanout.

**Temporary queues

**Bindings

We've already created a fanout exchange and a queue. Now we need to tell the exchange to send messages to our queue. That relationship between exchange and a queue is called abinding .

 

Sample 4: Routing

In the  previous tutorial   we built a simple logging system. We were able to broadcast log messages to many receivers.

In this tutorial we're going to add a feature to it - we're going to make it possible to subscribe only to a subset of the messages.

 

We will use a  direct   exchange instead. The routing algorithm behind a  direct   exchange is simple - a message goes to the queues whose  binding key   exactly matches the  routing key   of the message.

 

Binding Key:

channel.queueBind(queueName, EXCHANGE_NAME, "black" );

Routing Key:

 channel.basicPublish(EXCHANGE_NAME, severity , null, message.getBytes());

 

Sample 5: Topic exchange

Although using the direct exchange improved our system, it still has limitations - it can't do routing based on multiple criteria.
You might know this concept from the syslog unix tool, which routes logs based on both severity (info/warn/crit...) and facility (auth/cron/kern...).

Messages sent to a topic exchange can't have an arbitrary routing_key - it must be a list of words, delimited by dots.

 

* (star) can substitute for exactly one word.
# (hash) can substitute for zero or more words.

*.orange.*

 

Sample 6: Remote procedure call (RPC)

 

But what if we need to run a function on a remote computer and wait for the result? Well, that's a different story. This pattern is commonly known as Remote Procedure Call or RPC.