goahead-overview

p { margin-left: 1in; margin-bottom: 0in; }h1 { margin-bottom: 0.08in; }h1.western { font-family: "Liberation Serif",serif; font-size: 18pt; }h1.cjk { font-family: "DejaVu Sans Condensed"; font-size: 18pt; }h1.ctl { font-family: "DejaVu Sans Condensed"; font-size: 12pt; font-weight: normal; }h2 { margin-bottom: 0.08in; }h2.western { font-family: "Liberation Serif",serif; font-size: 14pt; }h2.cjk { font-family: "DejaVu Sans Condensed"; font-size: 14pt; }h2.ctl { font-family: "DejaVu Sans Condensed"; font-size: 12pt; font-weight: normal; }h3 { margin-left: 1in; margin-bottom: 0.08in; }h3.western { font-family: "Liberation Serif",serif; font-size: 12pt; }h3.cjk { font-family: "DejaVu Sans Condensed"; font-size: 12pt; }h3.ctl { font-family: "DejaVu Sans Condensed"; font-size: 12pt; font-weight: normal; }a:link { color: rgb(0, 0, 255); }

Functionality Overview of an Open Source Embedded Web Server

 

The dramatic growth in Internet appliances and devices has been directly due to drastically cheaper hardware, powerful 32 bit operating systems and the ubiquity of the Internet. As the Internet continues to grow, there will soon be more appliances and devices connected to the Internet than there are people using the Internet. In addition, consumers are beginning to expect devices that can “talk to each other.”

Consequently, there is a strong drive for an embedded Web server to use in all these devices. It is essential that this Web server be open and available to all. Open source Web servers permit hardware designers to easily integrate a well- tested and proven embedded Web server solution. This method leverages the effort of a large group of developers to strengthen the code base.

This White Paper provides an overview of the requirements for open source embedded Web servers, including GoAhead® WebServer™, and the design requirements and issues involved when implementing industry standards in embedded Web servers.

 

 

 

 

GoAhead Software, Inc

10900 NE 8^th Street, #750

Bellevue, WA 98004

425-453-1900

www.goahead.com

info@goahead.com

 

 

 

 Copyright 1999, GoAhead Software Inc. All rights reserved

 

GoAhead® is a registered trademark. All products identified by ™ in this document are trademarks of GoAhead Software, Inc. All other brand and product names are trademarks or registered trademarks of their respective holders.

 

Technical and product information in this document is subject to change without notice.

 

 

Revised: May 2000

Contents

 

 

Functionality Overview of an Open Source Embedded Web Server 1

Introduction 1

The Power of Ubiquity 1

Strength in Open Source 2

Server-based vs. Embedded Web Servers 3

Requirements of Embedded Web Servers 4

Small Memory Footprint 4

Security Model 4

Dynamic Web Pages 5

In-process CGI 8

References 12

Useful Sources 12

Introduction

As appliances and devices are connected to the Internet, they are becoming increasingly powerful and complex – and are rapidly becoming the most significant computers in contemporary society. New appliances with real-time operating systems, Internet protocols and device-based applications are appearing daily and the future of a Post-PC era looks bright. However, as such appliances become even more widely pervasive, there will be demand for integrated management of multiple devices and services. Devices need a universal method for access and control. By using a browser as the client, a device can provide remote access and control through an embedded Web server.

However, most Web servers are not designed to be embedded and most commercial embedded Web servers use proprietary schemes to render dynamic data. What is needed is a powerful, secure, standards-based, embedded Web server designed on open source philosophies (like Linux).

This paper will outline the design of such an embedded Web server, and will cover standards such as embedded JavaScript and embedded Active Server Pages for creating vivid, dynamic embedded applications.

The Power of Ubiquity

The law of “increasing returns" dictates that as a resource is more widely shared and consumed its value is enhanced rather than being diminished. This counter-intuitive law has been proven again and again by modern technology, but never more emphatically than by the Internet that becomes more valuable as increased users and publishers share a common resource.

Embedded devices and systems become more useful when users can control any and every device via a common user interface: the browser. While some market segments (data communications) have deployed embedded Web servers, the vast majority of embedded systems remain isolated and cannot be remotely accessed via a browser. There is great value in having a Web server in every device: in every phone, TV, VCR, security system--in every home and Internet appliance. When a critical mass of installed embedded Web servers exists, then we can derive the most value from appliances and embedded systems by being able to interconnect and control them remotely. It is expected that by the year 2003 there will be more appliances and devices connected to the Internet than there are people.

Strength in Open Source

Open source is a recent evolution of the long-standing free software tradition. It has gained considerable momentum. The vital essence of open source is that open access to source code enables developers to evolve software in new and interesting directions. According to the Open Source Initiative (http://www.opensource.org ), “Open source promotes software reliability and quality by supporting independent peer review and rapid evolution of source code. To be certified as open source, the license of a program must guarantee the right to read, redistribute, modify, and use it freely.” Bugs can be fixed, new features can be added and it can be improved to meet new challenges. This happens via the Internet at a very rapid pace. For example, since GoAhead posted the GoAhead WebServer™ 12 months ago, the public developer community has contributed nine known ports of the software to HP-UX, Windows CE, pSOS, QNX, IRIX, uCOS, eCOS, chorus and RTEMS.

For this paper, Web servers are classified into two categories: server-based and embedded. Many server-based Web servers are now provided under an open source banner. Linux, the Apache Web server and other companies are establishing organizations to meet the service and support requirements of commercial deployment of open source solutions.

Our goal in releasing the GoAhead WebServer under an open source license is to:

• Help create an embedded Web server implementation that can be used in every device.

• Harness the pool of talented developers to enhance the Web server in new directions.

• Enhance the reliability of the Web server by leveraging the test and debugging efforts of thousands of companies.

Server-based vs. Embedded Web Servers

At last count (http://webcompare.internet.com) , there were 41 Web servers listed by WebCompare. Although more do exist, only 5 were classed as embedded Web servers. So what is the difference between an embedded Web server and a server-based Web server? A server-based Web server has a very different set of requirements and is targeted to meet the needs of sites that publish World Wide Web server-based pages and/or host electronic commerce. For example, some of the requirements for a server-based Web server include:

• Can write to multiple logs

• Supports Virtual Servers

• Comes with SNMP agent

• Supports SSL v. 3

• Integrated certificate server

• Remote maintenance

• Search engine

These requirements are vital for high-volume, server-based Web servers but are of far less value to embedded Web servers that power Internet appliances. Embedded Web servers are more concerned with memory footprint, ability to generate dynamic data and the ability to port and integrate source code into a very customized device.

Some may argue that because a few of these server-based Web servers do provide source and may be free, they can be adapted to suit embedded needs. But the large memory footprint and resource requirements of say Apache, render them useless in most embedded systems. According to the Apache Project, Apache is “a collaborative software development effort aimed at creating a robust, commercial-grade, featureful, and freely-available source code implementation of an HTTP (Web) server.” While Apache focuses on being “featureful”, the needs of the embedded community definitely do not need such expansive feature sets or the “feature-creep” that characterizes many enterprise applications.

It is however, instructive to learn from what has made Apache the most successful server-based Web server today. Aside from a well-proven technology, it is the open development process that has made Apache what it is today and it is this same process that is required to create a standard embedded Web server that is designed specifically for Internet appliances and embedded systems.

Requirements of Embedded Web Servers

Internet appliances and embedded systems need Web servers to enhance existing functions without impinging on vital device functionality. As many of these devices are very cost constrained, memory and CPU resources are often at a premium. It is vital that embedded Web servers are very efficient and sparse in their memory requirements. The requirements of an embedded Web server include:

• Small memory footprint

• Configurable security model

• Strong support for dynamic generation of Web pages

• Easy device integration

• Support for devices without a file system

• Portable to new platforms and CPU architectures

Small Memory Footprint

A small memory footprint is one of the most vital requirements. Not only must the Web server use little memory (code, heap and stack); it must not fragment memory. Many embedded devices have basic memory allocation schemes that do not coalesce memory effectively. Even if good memory allocation mechanisms do exist, Web servers may cause memory to be fragmented by competing for memory with other embedded applications – especially since Web servers often have to respond to rapid requests to serve pages. Once the memory used to serve a page is freed, it may be useless, as it cannot be merged with adjacent memory blocks in the heap. To solve this problem, embedded Web servers must be fixed in the memory they use. Pre-allocation or static allocation schemes can be used to solve the problem.

Security Model

Embedded servers on remote devices obviously have important security needs. Security models (all supported by GoAhead WebServer) include:

Secure Socket Layering (SSL) for authentication and encryption over TCP/IP networks. SSL is essentially invisible to the end user: the Web browser might use a visual cue on the device to indicate duration that SSL is active, or may notify the user that they are about to enter a secure Web page.

Digest Access Authentication (DAA), a more secure authentication scheme for HTTP in which passwords are always encrypted.

User management functions that allow different users to have different levels of access.

Dynamic Web Pages

Unlike server-based Web servers, most Web pages generated by Internet appliances and embedded devices are dynamically generated. Whereas a server-based Web site will usually have vast stores of static pages, an embedded device will generate pages on demand. These pages display device status, results of sensors or local data that has been captured by the device.

Easy Device Integration

Hardware engineers today have unrelenting pressure to come to market early. Consequently it is vital for embedded Web servers to integrate easily. There are two parts to integrating a Web server:

• Integrating the Web server application into the Real Time Operating System.

• Integrating device access functions into the Web server.

Integrating a Web server into an application is most easily accomplished when source code is available. Often device applications require that the Web server be integrated into an existing application or that the event loop of the Web server be accessible. If the Web server is supplied only in binary form, or is costly to acquire in source form, such integration is difficult.

Integrating device access functions into a Web server requires easy binding of existing device APIs to URLs. This can be accomplished in many ways, but usually it is in a proprietary device specific method dictated by the Web server vendor. What is preferable is an industry standard method of linking the device to the Web. Active Server Pages and JavaScript are such standards that enable dynamic data from device functions to be linked to Web pages for easy access.

Support for Storing Web Pages in ROM

Many devices do not have disk or permanent storage yet still wish to be accessed and controlled via the Web. For such devices, a method of storing Web pages in ROM is required. Embedded Web servers should be able to compile Web pages and then link the page object into the resulting executable. Server-based Web servers have no such requirement.

Portability

As device engineers often move from one hardware project to another, it is important that the embedded Web server be highly portable. Experience should be transferable from one project to another. Source code that has already been ported and tested to a wide range of platforms is required.

Creating Dynamic Data

In an embedded device, the majority of Web pages are often dynamically created. As the primary function of the Web server is to permit access and control, it is important to display the device’s current status and provide operational feedback.

There are two primary ways to create a dynamic Web page:

• Generate the HTML tags via conventional C code

• Create the Web page and insert dynamic data via expansion tags

Generating HTML tags via C code requires no special handling by the Web server and may seem an attractive approach at first. However it does not take long before the “edit, compile, display, re-edit, re-compile, re-display” cycle becomes very tedious. Web pages are hard to engineer if you cannot see the final result. Manually generating the HTML tags prevents visualization until the page is actually generated in the device.

• The second approach allows much faster development cycles. Leading HTML design tools such as DreamWeaver can be used to create Web pages in a WYSIWYG manner. All that remains, is the dynamic data that is replaced at run time.

Active Server Pages (ASP)

Most embedded Web servers use proprietary schemes for displaying dynamic data. Some use proprietary tags that are expanded by the server before returning pure HTML to the browser. Others create new languages that contain mini-tags that are expanded into real tags with the dynamic data. All these schemes are adequate and have certainly done the job. However, if there is a more standards based approach that can be employed it should seriously be examined.

Active Server Pages (ASP) is a Microsoft developed approach to allow the easy creation of dynamic Web pages. Originally shipped in Microsoft IIS, it has now been ported to a wide variety of platforms and is available from many vendors in commercial products. There are many books about ASP as any trip to Barnes and Noble will testify, and there are now engineers with strong experience using ASP.

Active Server Pages permits the scripting of dynamic data using JavaScript or any other supported scripting language. The Web server would then evaluate the ASP script and the results are substituted into the page replacing the original script. This should be done in a one-pass operation for maximum efficiency. By using such server-side scripting, the dynamic data to be displayed is easily modified without recompiling the Web server. This results in very rapid development cycles.

Web pages using ASP normally have an “.asp” extension to distinguish them form normal HTML pages. To insert ASP scripts in a Web page, the scripting code is encapsulated using the special tags “<%” and “%>”. For example:

<H2>Today is
<% write(Date.tostring()); %>
</H2>

Various scripting languages may be selected at run-time by specifying the desired language:

<% language=javascript
write(“hello world”); %>

However, multiple-scripting languages that may be common in server-based Web servers would be rare for embedded Web servers.

In-process CGI

Common Gateway Interface (CGI) implemented as an in-process forms execution is another method of supporting dynamic content.

Old style CGI processing results in the creation of a new process for each and every request to a CGI URL. As CGI is often the primary means of implementing user input, this is very slow and cumbersome. An implementation of in-process CGI called GoForms is used in GoAhead WebServer and is a more suitable solution for embedded systems which demand compact, high performance solutions.

GoForm procedures run in-process without creating a new excecution for each browser connection. By sharing the address space with the GoAhead WebServer, GoForms procedures can directly access the full request context. The GoForm handler also automatically decodes and parses all POST and query data for easy access. A convenient API permits easy access to CGI variables.

GoForms is implemented as a URL Handler which interprets URLs that begin with "/goform". The URL segment following "goform", defines the form name with query details appended to the URL. For example:

/goform/myForm?name=John&age=30

This calls the GoForm procedure "myForm" and automatically decodes the querystring"name=John&age= 30" and defines GoForm variables called "name" and "age". In addition to POST and query data, the standard CGI variables are also defined.

GoForm procedures are defined by the websFormDefine API call and conform to the following prototype:

 

void myForm(webs_t wp, char_t *path, char_t *query)

{

websHeader(wp);

websWrite(wp, "Name %s", websGetVar(wp, "name", ""));

websWrite(wp, "Age %s", websGetVar(wp, "age", ""));

websFooter(wp);

websDone(wp, 200);

}

The GoForm procedure is responsible for writing the HTTP header and HTML document contents back to the user's browser. websHeader creates a standard HTTP header with initial HTML tags. websFooter closes the document with a closing HTML tag. Inside the GoForm procedure, websGetVar, websRedirect, websWrite, and websWriteBlock are some of the most useful API calls.

Embedded JavaScript

JavaScript has gained widespread attention as a leading scripting language, but unfortunately its large memory footprint (200 – 400K) has prevented its use in embedded applications. There is a compromise solution that has proven very effective. That is, to implement a strict subset of JavaScript called Embedded JavaScript. When used for browser-side scripting, complete access to the browser Document Object Model (DOM) is highly desirable and many of the features of JavaScript have evolved to service this need. However, on the server-side, Embedded JavaScript does not need the complete feature set, but rather basic control-flow, variables, and testing constructs. For example: the GoAhead WebServer implements the following subset of JavaScript:

 

Variables and variable declarations (var)

Global functions with support for local JavaScript variables

If, then, else

For loops

Expressions including: + - / * % ++ -- | & || && << >> < <= == > >=

Statements: return, braces, comma, semicolon

Comments

Missing from the embedded implementation are arrays, objects, regular expressions and object methods. The resulting implementation is a 15K Embedded JavaScript interpreter that is highly tuned to the needs of embedded devices, rather than a 200K conventional memory footprint.

JavaScript and ASP can be implemented efficiently and can execute very quickly. For example, the GoAhead WebServer can render over 60 ASP JavaScripted pages per second on a 266 MHz Intel x86 CPU. A tight recursive decent JavaScript parser and a hashed symbol table lookup for the variable store enable very high performance with minimal CPU requirements.

This feature set permits powerful server-side scripting to create dynamic Web pages. The following is an ASP / JavaScript example to loop over a set of sensors and output their values:

<H1>Sensor Values</H1>
<%
for (x = 0; x < 10; x++)
write(“<H2> Sensor ” x + “ is “ +
getSensor(x) + </H2>”);
}
%>

JavaScript is an untyped language and simply assigning a value to a variable will declare it. For good practice, the var statement can be used to declare variables. This lack of typing should not be a quality issue if the size of the scripts is kept small. JavaScript is not a replacement for C and does not scale well to implement large projects. It does however, excel at localized scripting to create dynamic content.

To ensure the embedded Web server can be easily extended and to provide easy binding of C code to JavaScript, a method of publishing C functions should be possible. The JavaScript standard includes both object methods and global functions. However, for Embedded JavaScript, it is the global functions that are both easiest to implement and most useful. Global functions can provide direct linkage from JavaScript to C functions. For example:

<% write(myJavaFunction(arg1, arg2)); %>

To publish the C function that implements this global JavaScript function, code similar to the following would be used:

ejSetGlobalFunction(handle, “myJavaFunction”,
cFunction);

When “myJavaFunction” is called via the ASP script interpreter, the JavaScript interpreter looks up “myJavaFunction” and locates the previously published function. The C function may return data by calling:

ejSetResult(handle, “any return data”);

By designing appropriate JavaScript global functions, a single ASP / JavaScript call can be used to output an entire table or return device or database information.

URL Handlers

The URL name space manages to encode the entire World Wide Web of information into a single globally unique addressing scheme. Often this becomes quite convoluted, as any query to a major search engine will demonstrate:

http://www.altavista.com/cgi-bin/query?pg=q&
kl=XX&q=embedded+AND+web+AND+servers

Leading Web servers leverage the URL name space to provide a better, more intuitive naming convention within the URL space. This is achieved by binding a URL prefix to a processing function that can handle the request. For example:

/thermostat/setTemp/72

This would invoke the thermostat handler that would parse the rest of the URL and set the temperature to 72 degrees.

Run Time Support

To round out a solid embedded Web server, the core product should be not be implemented in a “hard-coded” fashion that is difficult to extend. The GoAhead WebServer includes a memory allocator to reduce memory fragmentation, track memory leaks and check memory integrity; a hashed symbol table store; and a ring queue storage data structure. These modules provide a platform for easy application extension.

Open Source Development Model

To effectively support users, the open source development model needs two key ingredients:

• A vibrant development community with effective means of communication.

• Direct involvement in communications forums by the original author.

Developers need to be able to contribute and have their enhancements recognized. News groups, email updates and an effective Web site are vital.

The ongoing direct involvement of the original author is also vital to provide guidance and editing of the master source. This may, over time, be subsumed by a team of developers outside the author.

References

[GAEJS] GoAhead Software, “GoAhead Embedded JavaScript Specification”, April 1, 1999.

[XMLW3C] Extensible Markup Language, http://www.w3.org

[SGMLISO] Standard Generalized Markup Language, ISO 8870:1986(E).

[ECMA-262] JavaScript ECMA-262, http://www.ecma.ch

[JSPR] David Flanagan, "JavaScript: Pocket Reference", 1998, O’Reilly & Associates, Inc.

[GAEMFTR] "GoAhead Embedded Management Framework Technical Reference" http://www.goahead.com , GoAhead Software, Inc.

Useful Sources

The ASP Toolbox, http://www.tcp-ip.com/

Active Server Pages.com, http://www.activeserverpages.com/

The source, Microsoft’s IIS home page, http://www.microsoft.com/ntserver/web/default.asp