使用RabbitMQ使得Tomcat Sessions 变得更加云化

原文 http://www.jdon.com/jivejdon/thread/39106

 

Clustering Cloud-friendly Tomcat Sessions with RabbitMQ: Part I | TomcatExpert

云架构的流行，使得水平伸缩成为王者( horizontal scalability is king)，而传统的Tomcat集群是通过广播集群 ，也就是让一台机器上的Session广播到其他机器上，这样的坏处就是大量Session中数据在服务器之间不停被复制拷贝，增加服务器负担，服务器越多，复制越厉害，Tomcat的服务器也就不能无限任意增加，水平伸缩很差。

该文作者通过引入RabbitMQ消息系统，用来控制tomcat的Session无限复制，也就是设计一种Session管理器:使得任何一个用户请求都可以被随机分配到任何一台tomcat服务器上。

作者对Session管理器要求是：
1.轻量，Terracotta虽然号称可以帮助session复制，可惜重量。作者没有选择。
2.Reliable 可靠
3.Scalable 可伸缩
4.No single point of failure 无单点风险
5.Asynchronous (for speed and efficiency) 异步
6.Easy to configure易于配置
7.Zero (or nearly so) maintenance 零维护。

Session管理器的工作原理如下：当一个tomcat服务器被要求加载用户的session, 它首先将session的ID传到云存储中CloudStore(使用 Redis 或其他 NoSQL 数据库), 检查内部的session ID列表. 如果发现存在这个ID, 就知道这是一个已经激活的session，然后检查内部Map看看其是否已经存在当前服务器本地了，如果没有，发出"load"消息给相应队列 queue. 当前用户请求的线程就堵塞等待Session在本地加载复制。

[该贴被banq于2010-06-22 10:55修改过]

 

Clustering Cloud-friendly Tomcat Sessions with RabbitMQ: Part I

posted by jbrisbin on June 21, 2010 07:11 AM

The existing solutions for Tomcat session clustering are viable for moderate clusters and straightforward failover in traditional deployments. Several are well developed with a mature code-base, such as Apache's own multicast clustering . But there's no getting around the fact that today's cloud architectures place new and different expectations on Tomcat that didn't exist several years ago when those solutions were being written. In cloud architectures, where horizontal scalability is king, a dozen or more Tomcat instances is not unusual.

In my hybrid, private cloud, first described in my earlier post on keeping track of the availability and states of services across the cloud , I wanted to fully leverage all my running Tomcat instances on every user request. I didn't want to use sticky sessions as I wanted each request (including AJAX requests on a page) to be routed to a different application server. This is how virtual machines work at a fundamental level. Increased throughput is obtained by parallelizing as much of the work as possible and utilizing more of the available resources. I wanted the same thing for my dynamic pages. Not finding a solution in the wild, I resolved to roll my own session manager. The result is the "vcloud" (or virtual cloud) session manager. I've Apache-licensed the source code and put it on GitHub--mostly to try and elicit free help to make it better. You can find the source code here: http://github.com/jbrisbin/vcloud/tree/master/session-manager/

The Dilemma

I dug into the code of the available session managers and found they simply would not do what I was asking them to do. My expectations were (and are) quite high. I was looking for something that was, simultaneously:

1. Lightweight (sorry Terracotta)
2. Reliable
3. Scalable
4. No single point of failure
5. Asynchronous (for speed and efficiency)
6. Easy to configure
7. Zero (or nearly so) maintenance

I don't have to give the standard session managers a second thought when deploying a web application onto Tomcat. I wanted the same out of any other session manager. I also didn't want user sessions "copied" throughout the network all the time. I use lots of little Tomcat instances that have small heap sizes. If there was any way around it, I didn't want to keep entire copies of every active session in every server. But if I wanted the session loaded on demand, I'd need something screamingly fast. I use a cluster of RabbitMQ servers internally, so it only made sense to leverage that great asynchronous message broker to solve this problem.

Ideally, a user session would always be available in-memory. This is the fastest possible way to serve the user request. But it's inherently limiting. What if a new server comes online? Or, why keep precious resources locked up caching user sessions if that application server never sees that user?

The Trade-Off

Ain't that just life? Always having to trade one good thing for something else that you hope will really be better. Having user sessions available on-demand in any server that needs them is definitely trading off the speed of having a user session in-memory with the flexibility of not knowing (or caring) where that session object comes from. But realistically, it's only trading off the time it takes to handle the bytes that make up the object going from one server to the next. Since all my servers are connected by either a VMware virtual switch (because they're running on the same host) or are directly connected to our core Cisco gigabit switch; and because the only object stored in those user sessions is a Spring Security user object, I felt okay taking that hit on page load time if it meant I could load-balance all my servers, all the time.

The Method Behind the Madness

As a Java programmer, I found myself doing what nearly every Java programmer alive does: I tried to complicate things by adding to the code all kinds of stuff I didn't need. It took me a while to filter that out but I eventually settled on only the components that are actually required: a fast, thread-safe list of session IDs that are currently active in the cloud, and a convention of subscribing and publishing standard messages to queues with names that include the session ID.

The Process (at 30,000 feet)

When a Tomcat server is asked to load a user session, it passes that ID into the CloudStore, which checks the internal list of session IDs. If it finds that ID listed there, it knows it has an active session and checks the internal Map to see if that session is actually local to the application server. If it's not, it publishes a "load" message to the appropriate queue. The thread is then blocked for a configurable amount of time until it's either got the loaded session or it times out and assumes the session is no longer valid (maybe because the server that had it in-memory crashed).

As a somewhat-related aside: the Tomcat server doesn't know what's on the other end of that queue. It could be another Tomcat server that has that session in its internal Map. It could also be a dedicated session replicator written in Ruby and backed by a Redis (or other NoSQL) store. Going the other direction: code running in standalone mode, like a system utility, could load the session object that was created when a user logged into your initial web application, exposing a single user object to any Java code running anywhere in your cloud. Pretty cool, huh? ;)

When the server responsible for that session gets this load message, it serializes the session it has in-memory and sends those bytes back to the sender. The sender's listener gets this "update" message and checks to see if any loaders are waiting on this session. If it finds one, it gives the deserialized session to the loader, who, in turn, passes the session back to Tomcat, who continues to serve the page.

The Code (at 1,000 feet)

We won't have time in just one article to cover the important parts of the CloudStore. In my next article, I'll discuss the "load" and "update" event handlers in detail as well as issue the obligatory caveats and acknowledge the known shortcomings. But to start with, let's take a quick peek at the event dispatcher, which is the backbone of the CloudStore session manager.

There are several different programming models when writing asynchronous applications. There's no one way to tackle the problem, so it often comes down to simply personal taste. I can get my mind around the dispatching model more easily than some of the alternatives, so that's the route I chose. Fundamentally, the dispatcher has a custom EventListener class that runs in a separate thread which is responsible for pulling messages off the queue, interrogating them, and dispatching them to the appropriate handler based on their type. There is a message type for every action the CloudStore performs.

• A touch message adds that session ID to the master list, which marks it as a valid session throughout the cloud. One of these is emitted whenever a new session is first saved by the session manager.
• A destroy message not only deletes the session object from the internal Map, it unbinds the queue for that ID. These are emitted by the store when the session has expired or is otherwise invalidated.
• A setattr message contains incoming changes made to the session while it was on the remote server. These changes have to be replicated to the real session object in the internal Map. One of these is emitted by the session whenever the "setAttribute" method is called with a different object than what the attribute is currently set to.
• A delattr message is emitted whenever the session's "removeAttribute" method is called. Its handler removes the attribute from the real session object.

Almost without exception, I dispatch EventHandlers into separate threads to do the actual work requested by the incoming message. I do this because I have only one QueueingConsumer per queue. I want as much throughput as possible, so the thread that pulls messages off the queue can very quickly get back to doing what it's supposed to concentrate on: processing messages as quickly as possible. Because I'm using the official RabbitMQ Java client , I have to do this in a separate thread so I can publish messages, bind and unbind queues, and perform other operations that require communicating back to a RabbitMQ server; otherwise, those operations would cause a deadlock if I tried to do them in the EventListener's thread.

The Teaser

The other event message types ("load" and "update") get emitted whenever the session manager attempts to load a session or when the server that has the actual object in memory responds to that load message, respectively. I'll cover those in my next article.

Jon Brisbin is an Architect/Analyst/Java Guru at NPC International, the world's largest Pizza Hut franchisee. He's been deploying web applications on Tomcat for over 10 years and currently focuses on cloud computing (virtual, hybrid, and private). He built a private cloud from scratch using VMware ESX, Ubuntu Linux, packing tape, and rusty baling wire. He's done consulting work with industry leaders and Mom-and-Pops alike. Prior to NPC, Jon developed new application frameworks, integrated the AS/400 with UNIX and Windows systems, developed Lotus Domino applications, hacked websites together with Perl CGI and a text editor, and served with US Air Force Intelligence in a very hot and sandy, but undisclosed, location. He lives in the rural Midwest.

He blogs on Web 2.0 (and sundry topics) on his website: http://jbrisbin.com/web2