HTTP Compression

The compression story

HTTP compression was introduced in the HTTP 1.0 specification (RFC 1945) and initially supported gzip and the UNIX compress algorithms x-gzip and x-compress. In the HTTP 1.1 protocol (RFC 2616), compression was further clarified and formalized, as demonstrated in these encoding schemes:

• Gzip. An encoding format produced by the file-compression program gzip (GNU zip) as described in RFC 1952. This format is a Lempel-Ziv coding (LZ77) with a 32-bit CRC (Cyclic Redundancy Check).
• Compress. The encoding format produced by the common UNIX file compression program compress. This format is an adaptive Lempel-Ziv-Welch coding (LZW).
• Deflate. The zlib format defined in RFC 1950 in combination with the deflate compression mechanism described in RFC 1951.

For HTTP compression to be used, both the browser and the Web server must support compression. Internet Explorer 4 and above supports HTTP compression, except on the Macintosh. Netscape 4.x supported compression, although there were reported bugs. Netscape 6 supports HTTP 1.1 and HTTP compression. Opera 5 and above supports HTTP 1.1.

Microsoft IIS 5.0, IBM HTTP Server, and Apache Web servers support HTTP compression.

==

HTTP Compression Overview

An HTTP request (from a browser) must specify that it accepts compressed content before the Web server can send the output back as compressed. Specifically, the Accept-Encoding header is used to specify that the browser can accept compressed output.

The code in Listing 1 is an example of an HTTP header in a GET request. In this example, the Accept-Encoding header tells the Web server it accepts gzipped output. If this header is absent, the Web server does not return compressed content in the response.

HTTP response

When the server sends back the resulting content, the Content-Encoding header reveals to the browser the format used to compress the content. Listing 2 is an example of an HTTP header in a response with compressed content. Note the Content-Encoding and Transfer-Encoding headers -- the content length is not specified in the HTTP header.

Listing 2. HTTP response headers

HTTP/1.1 200 OK Server: Microsoft-IIS/5.0 Date: Thu, 18 Sep 2003 17:29:44 GMT X-Powered-By: ASP.NET X-AspNet-Version: 1.1.4322 Set-Cookie: TEST=SESSIONKEY=Ac0641be0-be5e-4be6-8f1e-b4a8d2643e33; path=/ Cache-Control: no-cache, no-store Pragma: no-cache Content-Type: text/html; charset=utf-8 Content-Encoding: gzip Transfer-Encoding: chunked Expires: Wed, 01 Jan 1997 12:00:00 GMT Vary: Accept-Encoding

How Microsoft does it

The Microsoft IIS 5.0 Web server supports compression of dynamic and static content. When considering using dynamic compression, administrators should be familiar with some issues:

• Within the Knowledge Base article 272596, it states that when compression is turned on, the expiration date is set to January 1, 1997.
• Windows 2000 SP2 introduced a fix to support compression on a POST. Previously only GET requests supported compression (KB article 259760).
• By default, static compression only compresses htm, html, and txt file types and dynamic compression only compresses exe, dll, and asp file types. You must run a script to modify the IIS metabase to add additional types.
• Compression is handled through an ISAPI filter, which is installed at the WWW Service level and is configured for the entire Web server rather than for specific Web sites.
• The gzip (as specified in RFC 1952) and deflate (as specified in RFC 1951) compression schemes are supported (KB article 255951).
• Compressed static content will always be sent back instead of a "HTTP/1.1 304 Not Modified", unless Windows 2000 SP3 is installed (KB article 307633).

As an example, the file type aspx content (ASP.Net) is not compressed by default with dynamic compression so that file type must be added to the list (KB article 332603). To do that, you will:

1. Open a command prompt.
2. Type net stop iisadmin, then press Enter.
3. Type cd C:/InetPub/adminscripts, then press Enter.
4. Type the following, then press Enter:
   CSCRIPT.EXE ADSUTIL.VBS SET W3Svc/Filters/Compression/GZIP/HcScriptFileExtensions "asp" "dll" "exe" "aspx"
5. Type the following, then press Enter:
   CSCRIPT.EXE ADSUTIL.VBS SET W3Svc/Filters/Compression/DEFLATE/HcScriptFileExtensions "asp" "dll" "exe" "aspx"
6. Type net start w3svc, then press Enter.

Note that IIS will only compress dynamic content when the HTTP request headers sent from the browser contain Accept-Encoding: gzip.

Once you have configured IIS to compress ASP.Net content, you must enable it at the server level. To turn on compression in IIS, go to the IIS Server properties box and click Edit for the master properties of WWW Service. Then click on the Service tab and check the Compress application files box to enable dynamic compression. You then should restart the IIS server.

Back to top

How IBM does it

The job of performing HTTP compression is most often delegated to the Web server or a network device. For the IBM HTTP server and Apache Web servers, the mod_gzip module can be used. In the article "Speed Web delivery with HTTP compression," the use of this module is discussed.

Back to top

Tuning for performance testing tools

When running performance tests, do not assume the testing tool is sending the correct HTTP headers to the Web server to enable compressed content to be returned. In both Mercury Interactive's LoadRunner and Microsoft's Application Center tests, I found that by default, neither sent the Accept-Encoding request header to the Web server.

LoadRunner does not send the Accept-Encoding header in the HTTP requests unless you add the following line to the scripts: web_add_auto_header("Accept-Encoding", "gzip");. If this line is not added, then IIS does not return gzipped content.

In Application Center Test (ACT), the execution of a test script recorded in ACT uses HTTP 1.0 and it does not send Accept-Encoding request headers. ACT doesn't appear to have a global setting to switch to HTTP 1.1 so the test will sends the additional header.

When executing performance test scripts, don't assume that performance test tools are receiving compressed Web pages.

Back to top

HTTP compression performance test results

To give an example of compression performance, I tested IIS 5.0 dynamic compression using a typical application that accessed three Web pages. I recorded the sizes of the compressed and uncompressed Web pages. The tests used the Shunra Cloud product (which simulates a single network link on a computer, enabling users to test the performance of IP-based technologies under a variety of conditions such as latency, packet loss, jitter, and bandwidth constraints) to simulate 28.8K and 56K line speeds. The mathematical differences in response time, based on actual Web page sizes, are shown in the following table.

Table 1. HTTP compression performance

	Content Size in Bytes	Line Speeds (Kbs), Transmission Times (sec)
		14.4	28.8	56	128	256	348
Uncompressed	64975	35.25	17.63	9.06	3.97	1.98	1.46
Compressed	17478	9.68	4.74	2.44	1.07	0.53	0.39
Difference	47497	25.77	12.88	6.63	2.90	1.45	1.07
Savings (percent)	73.1

The following graph demonstrates that compression has a positive impact on transmission time for 128K line speeds or less for HTML content of this size. At higher line speeds, the question becomes whether the savings is noticed by the user.

Figure 1. The impact of compression on transmission time

The above table and graph illustrate the theoretical savings of using dynamic compression. My performance tests in the lab using Shunra Cloud at 28.8K and 56K had response-time results that matched the theoretical savings. It was expected that compression should consume more CPU resources on the Web server, but the lab tests indicated the impact to be minimal with a 2 to 3 percent increase in CPU usage.

The results shown above are attributable to line speed and Web page size, and so are not specific to IIS; the same results would be expected in applications executed from an open Web server.

Back to top

Alternatives to dynamic compression

In addition to IIS dynamic compression, a quick search on the Internet pulled up a number of products that support HTTP compression. They fall into two categories: ISAPI filters and external devices. The ISAPI filters are quite common, inexpensive, and possess various features for selectively compressing content. The external devices are typically network devices that offer HTTP compression as a feature.

Some of those alternative products are:

• PipeBoost is an ISAPI filter and costs $1,500 for an enterprise license.
• Packeteer AppCelera.ICX is a network device that sits between the load balancer and the Web server.
• NetScaler is a network device that provides functionality besides compression.
• HttpZip is an ISAPI filter.
• Note: Cisco has discontinued their Content Optimization Engine.

Back to top

A few more considerations

When the response time of Web pages becomes an issue, consider the following issues:

• Web-page transmission size over a slow line speed. You can reduce HTML page size by 70 percent when using dynamic compression. Compression does not affect HTTP headers; it only compresses the content.
• Secure Sockets Layer acceleration. If the Web site has many SSL-enabled Web pages, then use an external SSL acceleration device.
• Static page caching. Most HTML pages reference JavaScript (js), Cascading Style Sheet (css), XML, GIF, and JPG files. These files are cached by the browser after they are first accessed so the number of times the browser downloads the file is twice: once when it is first accessed, and once after it is modified. However, the browser will always ask the Web server for these files using a GET HTTP request and the Web server will respond with "HTTP/1.1 304 Not Modified". So traffic to the Web server always asks if these files have changed. Read on for more details.

Back to top

HTTP request and response headers for static files

For your reference, Listing 3 is a HTTP request for a static file and the HTTP response for a browser-cached static file. Note that the request headers are fairly large when compared with the response.

These HTTP request and reply headers occur for content cached in the browser. Suppose a Web page has 10 to 12 references to static files. A slow network connection still affects the response time since these request and reply headers can take several seconds to be transmitted. If the number of references to static files is high, then one should consider reducing the number of references if possible.

Back to top

In conclusion

If the network line between the Web server and the browser is slow, using dynamic HTTP compression yields positive transmission results for most Web sites with little impact on the server. If the Web pages being transmitted are large (greater than 100K), then compression might also be beneficial for faster line speeds. HTTP compression of static content, though, may not be as beneficial given that the browser can cache these files. But even if a static file is cached by the browser, an exchange between the browser and the Web server could slow response time over slow lines. In this situation, it is wise to reduce the number of Web page references to static content or consider a caching server that is close to the user.

Resources

• Visit WebReference for a detailed article on why to use compression to speed Web transmission.
  
  
• Read "Speed Web delivery with HTTP compression" to dissect Web compression, examine the benefits of HTTP compression, learn about several compression tools, and highlight the effectiveness of the technology in a case study (developerWorks, July 2003) .
  
  
• In "A data compression primer," read about the basic types of data compression, and get an introduction to the mathematics and algorithms that go into compression techniques (developerWorks, May 2001) .
  
  
• Reference the HTTP 1.1 protocols.
  
  
• Get information on WebSphere® Application Server and download the trial program of version 5.1 to try out.
  
  
• If you're already working in MS but want to include more open source options, J-ASP will migrate ASP apps to the J2EE platform.
  
  
• Browse for books on these and other technical topics.
  
  
• Check out the network devices and ISAPI filters mentioned in the article that are designed to enable Web-content compression:
  • PipeBoost was initially designed to work with ASP, but accelerates JSP as well.
  • Packeteer's PacketShaper Xpress network device.
  • NetScaler's network device.
  • HttpZip's ISAPI filter.
  • Note: Cisco's Content Optimization Engine has been discontinued.
  
  

About the author

		Alan Beal is a distributed computing specialist who specializes in WebSphere, middleware, and e-business architecture and development. He is an Application Development Analyst in the Technology and Operations Division at Bank One in Columbus, OH.