Automation for the people: Choosing a Continuous Integration server

A survey of open source CI servers: CruiseControl, Luntbuild, and Continuum

Paul Duvall , CTO, Stelligent Incorporated

Paul Duvall is the CTO of Stelligent Incorporated , which helps companies address software quality with effective developer testing strategies and Continuous Integration techniques that enable teams to monitor and improve code quality early and often. He is a contributing author to the UML™ 2 Toolkit and currently co-authoring Continuous Integration: Improving Software Quality and Reducing Risk (Addison-Wesley).

Summary:   With so many Continuous Integration (CI) servers to choose from, it can be difficult to decide which one is right for you. In the second article of the series Automation for the people , development automation expert Paul Duvall looks at a handful of open source CI servers, including Continuum, CruiseControl, and Luntbuild, using a consistent evaluation criteria and illustrative examples.

 

Off the top of my head, I can think of at least 12 CI servers, both commercial and open source, available on the market today. While they all seek to automate the process of building software, they each have their own associated pros and cons. Moreover, the myriad tools available have the unfortunate consequence of making the decision to pick one tool quite exhausting.

The thing to keep in mind is that when deciding to use a tool to automate a process, the tool really does matter . The wrong tool may limit your overall flexibility, take longer to perform simple actions, or lock you into certain supporting tools or processes.

Criteria for choosing a CI server

Often, the extent of an analysis of a new tool can be summed up as follows:

I heard Tim is using Acme Inc's tool and I think Tim is a smart guy. Therefore, I'm also going to use Acme Inc's tool. Now I am a smart guy.

About this series

As developers, we work to automate processes for users; yet, many of us overlook opportunities to automate our own development processes. To that end, Automation for the people is a series of articles dedicated to exploring the practical uses of automating software development processes and teaching you when and how to apply automation successfully.

If, instead you asked Tim why he chose to use Acme Inc's tool, you might discover it was because his company had mandated it. This is why it's important to analyze a tool based on your specific technical and, yes, political requirements. If you don't, you may end up with a tool that doesn't meet your needs, or even worse, a tool that doesn't help you.

A tool's features are typically a significant focus for most people when making decisions. Keep in mind, however, that while features play an important role, there are other criteria to consider. In my experience, I've found that five criteria are most helpful in evaluating a tool:

What is Continuous Integration?

Continuous integration (CI) is a practice that enables teams to receive feedback and make improvements on a continual basis rather than waiting until later in the development cycle to find and fix defects. Tools (like CruiseControl) run in the background polling a version control repository looking for changes. When a change does occur, the tool executes a predefined build script through Ant, for example. Continuous Inspection is facilitated by the practice of continuous integration

• Features
• Reliability
• Longevity
• Target environment
• Ease of use

And don't forget -- it's also important to examine these five traits objectively .

Product features

When it comes to CI server features, you should consider the tool's integration with version-control systems, its ability to handle build platforms like Ant and Maven, plus its capability to provide feedback and reports. Also don't forget to examine other features like build labeling and managing project dependencies. Lastly, it's helpful to understand a product's extensibility in case you need to make some specific enhancements.

Table 1 elaborates each feature in detail:

Table 1. Detailed CI server evaluation features

Feature	Explanation
Version-control system integration	If a tool doesn't support your particular version-control system, do you really want to write a custom integration for it?
Build-tool integration	In choosing a CI server, you need to consider which build tools you already use or will be using. For Java™ programming, there are a couple of clear favorites, Ant and Maven, and most all CI tools support them. If your build system isn't either Ant or Maven, does the CI tool support the ability to run a program from the command line?
Feedback and reporting	Consider the old adage, "If a tree falls in the forest, does anyone hear it?" If a build fails, does anyone hear about it? If no one does, what's the purpose of having a CI tool? All CI tools offer some notification mechanism, but which one will work best for you? E-mail? Instant messenger? RSS?
Labeling	Some development teams like to track builds by giving them unique labels so they can refer to a particular build instance at a later date. If this is important to you, be aware that only some CI servers provide this capability.
Project dependencies	In some cases, after you build one project, you may need to build another dependent project. Certain CI servers support this feature and some don't.
Ease of extensibility	How easy is it to extend the current functionality of the tool? Are there plug-ins that allow for simple extension or is it always a code change?

From a feature standpoint, the points noted above are critical in choosing the right CI server for your needs.

Product reliability

Because it's simple to download and use an open source CI server, you can try out the product to determine its reliability. Furthermore, there's often a correlation between a tool's reliability and how many years it has been on the market. With new products, you run the risk of undiscovered bugs. Also, user groups can be excellent resources for reviewing the problems issued against the tool. A large number of problem posts or an overabundance of complex questions may indicate an uphill battle with the tool.

Because the servers I cover are open source, you can usually find out how many people have downloaded them, which can also be an indication of a product's health. Few users can mean few feedback channels and might be a clue to look elsewhere.

Longevity prospects

Before downloading a CI server, it can be helpful to understand the future prospects for this server. Simply put, using a defunct tool or one heading for the graveyard generally is a bad idea. Check how many years a tool has been available and if there is a healthy amount of activity in the user groups. Just as you're able to gauge a product's reliability from user groups, an active community is a good sign of a tool's future prospects.

Target environment

CI servers do not work in all environments. You need to consider which operating systems the server supports, as well as the underlying system requirements. If the tool is written in the newest version of Python, for example, you need to verify that version of Python available for your OS.

Ease-of-use

A product's ease-of-use is probably one of the most subjective criteria to consider. Some people prefer to modify configuration files manually, while others want all administrative tasks performed from an application, such as a Web console. Some servers require that you click from one screen to the next to perform simple administration, while others provide intuitive wizards.

If you want to understand the nuts and bolts of a CI server, then snazzy administrative Web forms won't be important; however, if you are short-staffed and overworked you probably won't want to spend much time administering a CI server.

Keeping these five aspects we discussed in this section in mind, let's take a look at three CI servers: Apache's Continuum, CruiseControl, and the build management server Luntbuild.

---

Back to top

Apache Continuum

Continuum is one of the newest CI servers on the scene, and a respectable newcomer at that. The setup and configuration of Continuum is straightforward: simply download and extract a ZIP file, run a command program, and you're good to go. The Web-based interface makes it easy to configure a project. What's more, you don't need to install a Web server either because Continuum embeds the Jetty Web server. Furthermore, Continuum can run as a Windows service and also embeds context-sensitive documentation into certain sections of the application, which provides a great deal of help.

Want more detailed information?

With so many CI servers to choose from, this article gives you enough ammunition to investigate each of these servers in more detail and decide which is best for you. Because I am comparing three different servers, I am not going into the specific details of each server. I am just focusing on the options provided by these servers out of the box. If you need more information, consult the installation and configuration guides for each of the servers.

Easy does it

One of the first things you'll notice when using Continuum is its ease-of-use. You'll be able to get the server up and polling for changes on your project in a matter of minutes. In fact, getting started with Continuum on Windows only requires four steps:

1. Download the Continuum ZIP file (see Resources ).
2. Extract the file's contents to a local directory.
3. Run the run.bat file followed by the InstallService.bat.
4. Open a browser to http://localhost:8080/.

Continuum has built-in support for five version-control systems: Subversion, CVS, StarTeam, Bazaar, and Perforce. There is also partial support for other version control tools like Visual Source Safe and ClearCase. Continuum supports four build mechanisms as well: Ant, Maven1, Maven2, and Shell (command line).

Configuring Continuum

When you first access the Continuum Web application, you are defaulted to the guest account. The guest account gives you read-only access to all projects and has no capability to administer or configure any projects. It's easy, however, to create an Administrative user and then set some properties, which apply to all projects.

Figure 1 shows an administrative page, which provides you the capability to manage Continuum settings for all projects, including creating the Admin account and the working, build output, and deployment directories:

Figure 1. Configuring Continuum is a snap

Adding a project to monitor

Configuring Continuum to monitor a project couldn't be easier. It's as simple as selecting the desired build platform, such as Ant or Maven2, and then pointing Continuum to your desired version-control system.

Figure 2 shows the fields you'll need to populate to set up an Ant project:

Figure 2. Creating a project in Continuum

After saving this information, Continuum polls your version-control repository once an hour. You can modify the project settings to check more or less often. As we're talking about Continuous Integration here, I'd recommend checking for changes every five minutes instead of just once per hour.

By default, when using Ant, Continuum finds a project's build.xml file in the project's root directory. You can modify this setting if you are using a different name or it's not in the project's root directory.

Although Continuum is one of the newest players in the CI scene, it is making a significant impact with its ease-of-use and its support for many of the popular version-control systems and build tools available today. I expect to see the ability to add and view reports in future versions.

---

Back to top

CruiseControl

CruiseControl is the granddaddy of CI servers. It's been available for over five years, and in many ways, the CruiseControl server has become synonymous with the practice of Continuous Integration. In the interest of full disclosure, I should mention that I've been a CruiseControl user for years.

Improved installation

If it's been a while since you last tried CruiseControl and thought it was a burden to install and configure, you may want to have a look at a more recent version. Now, there are numerous ways to install CruiseControl. For example, if you are using Windows, you'll find the simplest approach is to download the binary executable and run it. Don't worry, you can still download the source, too.

Out of the box, CruiseControl comes preconfigured with a configuration file that polls a CVS repository and executes an Ant build script. You don't need to install a Web server either, as CruiseControl also embeds Jetty.

Polling a version-control system

Compared to Luntbuild and Continuum, CruiseControl leads the pack by providing support for well over 10 different version-control systems. Furthermore, CruiseControl provides support for many of the custom commands found in those tools. Listing 1 is an example of using the CruiseControl config.xml script to poll a Subversion repository:

Listing 1. Polling a repository via the config.xml file

<listeners> <currentbuildstatuslistener file="logs/${project.name}/status.txt"/> </listeners> <modificationset quietperiod="30"> <svn RepositoryLocation="http://www.qualitylabs.org/svn/ambientorb/trunk" username="bfranklin" password="G0Fly@Kite" /> </modificationset>

 

Executing a build script

When changes are detected in a version-control system, like Subversion, you can easily configure CruiseControl to execute a delegating build script. For example, Listing 2 demonstrates calling an Ant script from the config.xml, which instructs CruiseControl to poll the Subversion repository every 60 seconds and execute another Ant script. The delegating build script (not shown) deletes old files, checks out the latest source from Subversion, and runs the project's build script on the code.

Listing 2. Executing an Ant build script

<schedule interval="60"> <ant anthome="apache-ant-1.6.5" buildfile="build-${project.name}.xml"/> </schedule>

 

After you've set up this aspect of CruiseControl and start the server, you can access the CruiseControl Web dashboard as illustrated in Figure 3:

Figure 3. CruiseControl dashboard

CruiseControl dashboard

To receive feedback on the latest build, you can add an htmlemail plug-in to the config.xml script shown in Listing 3. You can use the config.xml file to configure many more feedback mechanisms too, such as sending text messages, electronic devices (via X10), and even instant messenger.

Listing 3. Sending an email with CruiseControl

... <plugin name="htmlemail" buildresultsurl="http://${env.COMPUTERNAME}/cruisecontrol/buildresults/${project.name}" mailhost="${smtp.server}" username="${mail.username}" password="${mail.password}" returnaddress="${buildmaster.email}" returnname="${buildmaster.name}" subjectprefix="${project.name} build" xsldir="webapps/cruisecontrol/xsl" css="${reportdir}/cruisecontrol.css"/> ... <htmlemail> <always address="${buildmaster.email}"/> <failure address="${buildmaster.email}"/> </htmlemail>

 

CruiseControl provides many useful features, has a strong user community, and is extremely extensible. As compared to the other tools evaluated in this article, some developers feel that CruiseControl is not as easy to use. On the other hand, some find that making changes with an XML script offers greater control.

---

Back to top

Luntbuild

In terms of years on the market, Luntbuild is the middle sibling between Continuum and CruiseControl. As compared to Continuum and CruiseControl, Luntbuild aims to be a build management server providing support for such things as parallel development and user group management. The entire configuration is managed through a Web application, so there are no configuration files to manipulate. There is a commercial edition available as well, called QuickBuild, which includes user support.

Jettisoned Jetty

Luntbuild offers a few installation approaches. You'll probably discover that the simplest approach is through the GUI installation. You configure and manage Luntbuild with the Web application; therefore, you'll need to ensure that you're running a Web server capable of handling JSPs, like Tomcat or Jetty.

Version control polling

Luntbuild provides support for eight different version-control systems such as CVS, Subversion, ClearCase, and Perforce. Figure 4 illustrates setting Luntbuild to poll Subversion:

Figure 4. Luntbuild polling a Subversion repository

Executing a build

Luntbuild supports five different build platforms, including Ant, Maven, Maven2, the command line, and rake (which is used for building Ruby applications). Figure 5 shows the configuration page for an Ant builder:

Figure 5. Executing an Ant script using Luntbuild

Build scheduling

By using the Schedule tab in Luntbuild, as shown in Figure 6, you can trigger how often Luntbuild polls the version-control system for modifications. On this tab, you can also designate the unique build label assigned to each build.

Figure 6. Scheduling a build in Luntbuild

Publishing results in Luntbuild

Once you've configured a project, a Version Control System adaptor, a build, and a scheduler, you can designate how users will receive feedback. Luntbuild, however, provides built-in support for e-mail and instant messenger only. In addition, you can monitor your builds from the main page of Luntbuild, as shown in Figure 7:

Figure 7. Monitor builds from the Luntbuild Web app

Luntbuild provides a host of powerful capabilities, including managing project dependencies and numerous Version Control System adaptors. I think that the workflow and user interface could be simplified because there are numerous steps you have to take to set up and configure the tool.

---

Back to top

A CI scorecard

Recommending which tool is right for you, without understanding your specific needs, would be reckless. Each of these servers has many excellent features, and as I mentioned at the outset, just because one of the CI servers works best for someone doesn't mean it will necessarily meet your needs.

If you're looking for a tool that is easy to use, check out Continuum. If extensibility, flexibility and a thriving user community are important to you, try CruiseControl. Look into Luntbuild if you're in search of Web administration and extended user support options. Development ecosystems have been built around each of these servers, so if there are certain features missing, you can usually find an extension to fit your needs.

In Table 2, I've summarized the five core aspects of features, reliability, longevity, target environment, and ease of use for each of the reviewed CI servers, based on my experiences with them:

Table 2. CI Server review aspects matrix

	Features	Reliability	Longevity	Target environments	Ease of Use
Continuum	Support for Ant, Maven1 and Maven2, and Shell. Remote management capabilities using XML-RPC and SOAP; support for Maven2; user groups; expect additional reporting and feedback mechanisms in the future -- without needing to make a code change.	Released in 2005. Expect to hear more on Continuum through its relationship with Apache.	Good user community support through Apache Maven. Product is still new on the market though.	Linux, Mac OS X, Solaris, and Win32.	Excellent ease-of-use and installation.
CruiseControl	Lots of version control integrations and extensibility. Remote access via JMX control. Multiple feedback mechanisms including RSS, X10, Jabber, and others.	Released in 2001. Of the three servers, CruiseControl is being used in most development shops.	Vibrant user community; every indication suggests CruiseControl will be around for a while.	Windows and Unix; any platform that can run a Java JVM.	Easy installation. Some find that they'd rather not modify XML configuration files.
Luntbuild	Project dependencies, labeling, security groups, and parallel development.	Released in 2004. Luntbuild provides extended user support options.	User community is not as active as CruiseControl.	Systems that can run a JVM and servlet container.	Easy installation, but the user interface/workflow could be vastly improved. Web-based configuration (no configuration files need to be modified).

Keep in mind, I've evaluated only three servers in this article; there are many more that may actually be better for your needs. Now that you understand how to pick a CI server though, the job should be easy. Stay tuned for next month when I'll cover build smells that are often committed on development projects.

 

Resources

Learn

• Automation for the people (Paul Duvall, developerWorks): Read the complete series.
  
• "Automate your team's build and unit-testing process " (Mark Wilkinson, developerWorks, October 2005): A guide through the practical issues involved in setting up a Linux-based build server for Java projects.
  
• "Realizing Continuous Integration " (Kevin Lee, developerWorks, September 2005): Learn the concept and practices of Continuous Integration.
  
• "Continuous Integration " (Martin Fowler, martinfowler.com, May 2006): A description of the purpose and practices of Continuous Integration.
  
• Practical Continuous Integration (Testearly.com): More detailed information on installing and configuring Continuum, CruiseControl, and Luntbuild.
  
• developerWorks : Hundreds of articles about every aspect of Java programming.
  

Get products and technologies

• Continuous Integration Server Feature Matrix : A matrix of many of the CI servers currently on the market along with their respective features.
  
• Continuum : Apache's foray into the CI server space.
  
• CruiseControl : A freely available open source server for Continuous Integration.
  
• Luntbuild : Another Continuous Integration server with extended commercial support.