Designing Data Storage Architecture-Azure Storage Options

http://www.infoq.com/news/2009/12/Azure-Storage

First some terminology. The term Windows Azure Platform covers all of Microsoft’s cloud offering. Within the Windows Azure Platform are three principal offerings. The virtual machine component is called Windows Azure. Also know as compute, it supports web roles for hosting web sites and worker roles for back-end processing. SQL Azure is based on SQL Server and has many of the same features. Finally there is Windows Azure platform AppFabric, which shouldn’t be confused with Windows Server AppFabric. Azure AppFabric is a message bus, with the goal of making authentication and message passing easier, especially across firewalls.

Azure Platform offers a plethora of storage options, making choosing which to use potentially quite difficult. At the core of Windows Azure is Blob storage. There are two types of blobs, block blobs and page blob. A bock blob can store up to 200 GB of data and is optimized for steaming. A page blob supports up to 1 TB of data and is meant for random access. In addition to being the foundation of many other features, blobs are meant to store resources such as images and videos that users can directly download using REST-style HTTP requests.

Running on top of page blobs is the Windows Azure XDrive. The XDrive can be mounted as if it were a NTFS formatted hard drive, allowing it to work with normal file I/O APIs. But since it is also a blob, it can be copied as an atomic unit.

If you are looking to store more structured data, you have a couple of options. For storing massive amounts of non-relational data your best bet is currently theTable Service. While each entity in an Azure Table can only be 1 MB in size, there is no hard limit to the overall size of a Table and it should scale into the terabyte range. Despite their large size tables are actually pretty restrictive, for they only support a handful of data types are require developers to think about partitioning from the very beginning.

If you want to use relational concepts like joins, then SQL Azure is a better route. SQL Azure supports most features you would expect from modern database including the standard ODBC and ADO.NET clients APIs. However it does have some serious size restrictions with the largest database being capable of only storing 10 GB. Partitioning across multiple databases wouldn’t be a big deal if it were not for the fact that cross-database queries are not supported, forcing any unions or joins to be done in memory.

Finally there are the message storage options. Message storage is meant for short term queuing and store-and-forward type architectures. Azure’s Queue Servicesupports XML-based messages of up to 8KB in size. If you need to send larger payloads the data should be stored elsewhere and referenced by URI or primary key in the message. Queue messages are “hidden” when they are received. Applications are expected to delete the message in a timely fashion; otherwise it will eventually become visible again and be redelivered. From a design standpoint this means that message processing has to be idempotent. Currently there is no hard limit on the number of messages in a queue.

Azure AppFabric also offers message queuing, though they call it the AppFabric Service Bus Message Buffer. The message buffer only supports 10 messages by default, but can be configured to store up to 50. Messages in the buffer are stored in memory and thus cannot survive a server crash, making them of limited use in situations where the receiving end may fall behind or go off-line for more than a few moments.