Sending large files over JMS

Sending large files over JMS

Posted by  guruwons on November 15, 2006 at 8:30 AM PST

JMS provides powerful asynchronous communication, but it is message-oriented and doesn't support InputStream/OutputStream at all. So it is nearly impossible to send a large file without loading it on memory. How can we acheive this? Let me talk an enhancement of JMS for this issue.

ActiveMQ provides InputStream/OutputStream extension on JMS Connection. And MantaRay JMS also supports similar InputStream/OutputStream facilities over JMS destinations. But these are not so good because they don't support JMS Session (this means they don't support transacted session either) and JMS Message in this case. It also looks awkward because it is far from message-based JMS API. It appears to me that it is impossible to handle both normal JMS messages and file on the same destination.

There is a better idea on JBoss forum. It is mentioning a FileMessage like below.

Interesting. In order to avoid storing the payload of the message in memory at any time, we could create a new type of message e.g. "FileMessage" which just contains a pointer to the file on disc. When the message is sent from client->server or from server node-> server node, the file is streamed rather than serialized, or the pointer is just passed if the filesystem is visible from both source and destination. Perhaps we could extend this technique to any large message. - Tim

Let me go further. FileMessage contains the URL of the file to send like below.

FileMessage

public interface FileMessage extends javax.jms.Message {
    /**
     * Returns the URL of the file.
     * The URL object is readable by <code>URL.openStream()</code>.
     *  
     * @return the <code>URL</code> of the file.
     */
    URL getURL();

    /**
     * Set the URL of the file. 
     * The URL object should be readable by <code>URL.openStream()</code>.
     * 
     * @param url the <code>URL</code> of the file.
     * @exception MessageNotWriteableException if the message is in read-only mode.
     */
    void setURL(URL url) throws MessageNotWriteableException;
}

Showcase

Let's see how we can send any file using above FileMessage.

    InitialContext ic = new InitialContext();
    
    QueueConnectionFactory qconFactory = (QueueConnectionFactory)ic.lookup("QueueConnectionFactory");
    QueueConnection qcon = qconFactory.createQueueConnection();

    // JEUS-specific Session
    JeusQueueSession qsession = (JeusQueueSession) qcon.createQueueSession(false, Session.AUTO_ACKNOWLEDGE);
    
    Queue queue = (Queue)ic.lookup("ExamplesQueue");
    QueueSender qsender = qsession.createSender(queue);

    // Create a FileMessage using JeusQueueSession
    FileMessage msg = qsession.createFileMessage();
    
    // set file URL and send the message
    File file1 = new File("/home/wonskim/temp/file1.jar");
    msg.setURL(file1.toURL());
    qsender.send(msg);
    
    qsender.close();
    qsession.close();
    qcon.close();

Senders just create a FileMessage and set the URL of the file. JMS runtime will read the content of the file using URL.openStream() and send it to JMS Server. There is no memory burden in clients.

Let's see receivers side. Receivers can be MDB or normal JMS Listener.

    public void onMessage(Message msg) {
        if (msg instanceof FileMessage) {
            URL url = ((FileMessage)msg).getURL();
        
            if(url != null){
                // read the content of the file
                InputStream inputStream = url.openStream();

                // do whatever you want...
                
            }
        }
    }

Receivers receive a FileMessage like other normal JMS messages. The URL value of the file is not same as what senders set. Receiver clients can just use the URL to get the content of the file like URL.openStream(). Cool!

Two Strategies

Actually there are two strategies in JMS runtime.

1. When JMS Server sends a FileMessage the content of the file is transferred together and JMS runtime stores the file in a temporary file. So the URL of the received FileMessage is the URL of the temporary file.
2. The content of the file is not transferred to receivers and the URL is a remote location on JMS Server. For example, http, https, ftp or custom protocol can be option.

I think the second is better because the content of the file is transferred only if it is required. There is one thing to consider - The remote URL will be valid only before the ack of the message is sent back to JMS Server. Because JMS Server can remove the file after the delivery of messages is confirmed. With AUTO acknowlege mode the ack will be sent after onMessage() finishes. So If we need to read the file, we should do it inside onMessage(). You can come up with how to do if you use other ack modes.

I've tried to implement this new type of message on TmaxAS JEUS(My company's Application Server). The result has been successful. What do you think of this idea?