Automation for the people: Continuous testing

Always run automated tests with every change to a code base

 

Running unit tests from an IDE like Eclipse or an Ant build script, for example, is a great start toward ensuring the quality of an application; however, running unit tests any time source code changes in a version control repository, like Subversion, on a separate, clean build machine can help verify problems throughout a development life cycle. Moreover, running different types of developer tests, such as component, functional and performance, can shine a light on problem areas earlier in the development life cycle.

Developer tests, which when run often in a Continuous Integration (CI) environment, effectively act like a spot light on code quality. This is because these tests, if written effectively, can find issues like defects almost as soon as they're created. Tests that aren't run often are less effective because the time between when a defect is coded and when it can be discovered is long, but running tests continually (that is, every time code changes) ensures that unintended behavior is discovered quickly.

This article covers the following:

• Running JUnit tests via Ant
• Executing longer running component tests using JUnit and DbUnit
• Determining which methods take too long to execute using JUnitPerf
• Running Web-based functional tests with Selenium
• Assessing code coverage with Cobertura
• Testing continually with CruiseControl

I provide an overview of the different types of developer tests along with examples that you can add to your build process and run continually using a Continuous Integration system.

Unit testing a la JUnit

You say unit test, I say component test

What is often referred to as a unit test is more likely a component-level test. A component test typically verifies more than one class and has a dependency on something like a database or other heavy mechanism, like a file system. Most importantly, however, on a test by test basis, component tests take much longer to run than a unit test.

Sometimes I hear developers lump the term developer tests as simple unit tests; however, I've found that it's helpful to refine the term unit testing as something more specific. To me, unit tests are fast running tests that typically test individual classes that don't have heavy external dependencies like databases. For example, Listing 1 defines a unit test that uses JUnit to verify a stubbed out data class called BeerDaoStub . The technique of testing against an interface that doesn't actually connect to a database, for example, is an approach that you can use to verify business functionality without incurring expensive setup costs. Plus, doing so keeps the test as a true unit test.

Listing 1. A simple unit test

public void setUp() { beerService = new BeerDaoStub(); } public void testUnitGetBeer() { Collection beers = beerService.findAll(); assertTrue(beers != null && beers.size() > 0); }

 

Once you've written a few unit tests, you can always run them through an IDE, but you'll also want to run them as part of a build process. Ensuring that tests run successfully through a build process means these same tests can be kicked off in the context of a CI build too.

Listing 2 is a snippet of an Ant script that demonstrates the junit task to execute a batch of unit tests. The beauty with this task, working in concert with JUnit, is that any tests that I've defined are now automatically run and if any of these tests fail, the build will fail also -- by using the haltonfailure attribute.

Listing 2. Running a unit test in Ant

<junit fork="yes" haltonfailure="true" dir="${basedir}" printsummary="yes"> <classpath refid="test.class.path" /> <classpath refid="project.class.path"/> <formatter type="plain" usefile="true" /> <formatter type="xml" usefile="true" /> <batchtest fork="yes" todir="${logs.junit.dir}"> <fileset dir="${test.unit.dir} "> <patternset refid="test.sources.pattern"/> </fileset> </batchtest> </junit>

 

Notice that test.unit.dir designates the location of the tests. This is an effective way to segregate these tests (unit, in this case) from other tests. By utilizing this technique, you can run faster tests first, followed by slower tests (such as component, functional, and system tests) by defining additional Ant targets.

---

Back to top

Collecting component tests

Because unit tests execute so quickly, they are easy to run often as a part of a build. These tests, however, don't achieve a high rate of code coverage -- their isolated nature means they only test a portion of the functionality. Writing tests that reach more code and therefore exercise more functionality usually takes more legwork in the form of adjunct frameworks. Once you start to use these helper frameworks to write tests, these tests start to become higher level ones that I tend to categorize as component tests.

Component tests are essentially tests that verify more than one class and, typically, rely on external dependencies such as a database. Component tests are written in much the same way as unit tests, except that instead of mocking or stubbing classes to force isolation, these tests bite the bullet, so to speak, and enlist frameworks to facilitate working with external dependencies. For example, I often employ the DbUnit framework to help manage a database so that my component tests can verify the functionality of code that relies on the database's data.

Controlling database state with DbUnit

DbUnit is a framework that makes the process of testing against a database much simpler. It provides a standard XML format for defining test data that can be used to select, update, insert, and remove data from a database. Keep in mind, DbUnit doesn't replace a database; it provides a more efficient mechanism for handling test data. With DbUnit, you can write tests that rely on specific data, which DbUnit ensures is present in an underlying database.

You can use DbUnit programmatically in JUnit or you can utilize it as part of a build process. The framework comes with an Ant task that provides a way to manipulate, export, or compare data in a database using XML files. For example, Listing 3 demonstrates the dbunit task, which, in my case, inserts test data into my target database and then removes the data after having run all my component tests:

Listing 3. Running a component test in Ant

<target name="component-tests"> <mkdir dir="${logs.junit.dir}" /> <taskdef name="dbunit" classname="org.dbunit.ant.DbUnitTask"/> <dbunit driver="com.mysql.jdbc.Driver" url="jdbc:mysql://localhost:3306/brewery" userid="${db.username.system}" classpathref="db.lib.path" password="${db.password.system}"> <operation type="INSERT " src="seedFile.xml "/> </dbunit> <junit fork="yes" haltonfailure="false" failureproperty="tests.failed" haltοnerrοr="true" dir="${basedir}" printsummary="yes"> <classpath refid="test.class.path" /> <classpath refid="project.class.path"/> <formatter type="plain" usefile="true" /> <formatter type="xml" usefile="true" /> <batchtest fork="yes" todir="${logs.junit.dir}"> <fileset dir="${test.component.dir}"> <patternset refid="test.sources.pattern"/> </fileset> </batchtest> </junit> <mkdir dir="${reports.junit.dir}" /> <junitreport todir="${reports.junit.dir}"> <fileset dir="${logs.junit.dir}"> <include name="TEST-*.xml" /> <include name="TEST-*.txt" /> </fileset> <report format="frames" todir="${reports.junit.dir}" /> </junitreport> <dbunit driver="com.mysql.jdbc.Driver" url="jdbc:mysql://localhost:3306/brewery" classpathref="db.lib.path" userid="${db.username.system}" password="${db.password.system}"> <operation type="DELETE " src="seedFile.xml"/> </dbunit> </target>

 

As you can see in Listing 3, my component tests can now rely on specific data residing in the database during their execution. What's more, this process is now repeatable because I've removed all the data after all tests successfully execute.

Seeding the database

You can use the INSERT and DELETE operation types for the dbunit task in conjunction with a seed file that contains XML elements representing database tables and associated rows. For example, Listing 4 is the contents of the seedFile.xml file referenced in Listing 3. Each BEER element represents a database table also named BEER and each of the BEER element's attributes and their values map to corresponding database column names and values.

Listing 4. A DbUnit seed file

<?xml version='1.0' encoding='UTF-8'?> <dataset> <BEER id='6' beer_name='Guinness Extra Stout' brewer='St.James Brewery' date_received='2007-02-01' /> <BEER id='7' beer_name='Smuttynose Robust Porter' brewer='Smuttynose Brewery' date_received='2007-02-01' /> <BEER id='8' beer_name='Wolavers pale ale' brewer='Wolaver Brewery' date_received='2007-02-01' /> </dataset>

 

As you may have noticed from Listing 3, you can reuse DbUnit's seed files for different operations. In my case, I used the file in listing 4 to seed the database before running my component tests and then used the same file to indicate which data to delete from the database upon test completion.

---

Back to top

Partaking in performance tests

Performance testing is often performed (no pun intended) long after developers have finished coding -- yet it's often the case that performance issues could have been found (and most likely solved) much earlier in the development cycle. Luckily, there is a way to solve this problem: continuous testing, or more specifically, continuously running JUnitPerf tests.

JUnitPerf is perfect for performance testing

JUnitPerf is a framework that works in concert with JUnit to execute test cases within a predefined time limit: If a method under test takes longer than a desired threshold, the test is considered a failure. By integrating performance tests into an automated build, you can effectively monitor application performance and even fail a build if performance issues pop up.

I prefer to use JUnitPerf as a simple way to discover early performance problems rather than as a mechanism to measure execution time though; tools like profilers are much more capable at providing this type of measurement. In essence, you can think of JUnitPerf as an early warning system.

In Listing 5, I define a JUnit test that uses JUnitPerf to verify the execution time of a test called testLongRunningMethod in the BeerServicePerformanceTest test class. If the test method happens to take longer than 1000 milliseconds to execute, it fails.

Listing 5. A performance-based test using JUnitPerf

package com.beer.business.service; import com.clarkware.junitperf.* ; import junit.framework.Test; public class ExampleTimedTest { public static Test suite() { long maxElapsedTime = 1000 ; Test testCase = new BeerServicePerformanceTest("testLongRunningMethod "); Test timedTest = new TimedTest(testCase, maxElapsedTime); return timedTest; } public static void main(String[] args) { junit.textui.TestRunner.run(suite()); } }

 

Be careful in using precise timings as a gauge for the execution time of a method; the setup and tear down time of a test is included in the overall execution time. Plus, precise measuring of execution speed is more of an art form than a science when it comes to early performance testing.

---

Back to top

Functional tests with Selenium

You can write all of the unit and component tests you want, but if you're writing an application that provides a user interface of some type (like a Web application, for example), you'll need to test the presentation layer. In the case of a Web application, you'll want to verify that the navigation of a user scenario and additionally verify that the functionality of the scenario pans out. Until recently, however, this sort of testing has proven to be a burden as typically, you'd need to purchase tools that promoted late cycle testing. What's more, these tools rarely fit into a build process even if the tests were built early enough.

Enter Selenium

In the last few years, however, a number of open source tools have sprung up that address functional testing; moreover, you can easily employ these tools early in the development life cycle. Tools such as Selenium and Watir are open source; what's more, they have been built with developers in mind. Besides programmatically defining Selenium tests in various languages (like Java™ programming and Python, for example), Selenium also provides an easy to learn table-driven format that can be used by non-technical types as well.

The Selenium framework uses JavaScript to execute Web-based acceptance tests that open a browser and run table-driven tests. For example, Listing 6 demonstrates an HTML table that represents a simple Selenium test. The various steps of the test open a Web application and then perform a login with a valid username and password. The test's results are generated in an HTML table that can be viewed after Selenium completes running all the tests.

Listing 6. Functional test with Selenium

<html> <head> <meta http-equiv="Content-Type" content="text/html; charset=UTF-8"> <title>MyTest</title> </head> <body> <table cellpadding="1" cellspacing="1" border="1"> <thead> </thead><tbody> <tr> <td>open</td> <td>/beer/</td> <td></td> </tr> <tr> <td>type</td> <td>username</td> <td>admin </td> </tr> <tr> <td>type</td> <td>password</td> <td>password</td> </tr> <tr> <td>clickAndWait</td> <td>//input[@value='Login']</td> <td></td> </tr> <tr> <td>verifyTextPresent</td> <td>Logged in as admin </td> <td></td> </tr> </tbody></table> </body> </html>

 

You can define multiple acceptance tests using the table-based format in Listing 6. You can also group tests into suites and execute an entire suite at a time.

Driving Selenium through Ant

What's great about Selenium is that it was created from the ground up with CI in mind because you can run Selenium tests from build tools like Ant. What's more, because of the forward thinking of the framework's designers, if any Selenium acceptance tests fail, you can also fail the entire build. For example, Listing 7 demonstrates an Ant task that uses the Selenium Remote Control server to execute a series of table-driven tests against a Web application:

Listing 7. Running Selenium with Ant

<?xml version="1.0" encoding="iso-8859-1"?> <project name="functional-tests" default="run-selenium-tests" basedir="."> <property file="${basedir}/selenium.properties"/> <import file="${basedir}/common-environment.xml"/> <property name="acceptance.test.lib.dir" value="${functional.test.dir}" /> <property name="firefox" value="*firefox" /> <property name="base.url" value="http://${web.host.name}:${web.port}" /> <property name="acceptance.test.list.dir" value="${functional.test.dir}" /> <property name="acceptance.test.report.dir" value="${functional.test.dir}" /> <target name="run-selenium-tests"> <mkdir dir="${reports.dir}" /> <java jar="${acceptance.test.lib.dir}/selenium-server.jar" fork="true"> <arg line="-htmlSuite "${firefox}""/> <arg line=""${base.url}""/> <arg line=""${acceptance.test.list.dir}/${test.suite}""/> <arg line=""${reports.dir}/index.html""/> <arg line="-timeout ${timeout}"/> </java> </target> <target name="stop-server"> <get taskname="selenium-shutdown" src="http://${web.host.name}: ${selenium.rc.port}/selenium-server/driver/?cmd=shutDown" dest="result.txt" ignoreerrors="true" /> <echo taskname="selenium-shutdown" message="Errors during shutdown are expected" /> </target> </project>

 

When executing Selenium tests, don't be alarmed when the framework opens a Web browser, executes your tests at lightning speed, and then closes the browser and generates an HTML report -- it's one more quick and easy way you can learn about problems early in the development life cycle (when they are easier to fix).

---

Back to top

Code coverage with Cobertura

To be at 100% or not, that is the question

When running tools like Cobertura or Emma, it is important to remember one important aspect: Just because you have 100% line coverage in a particular method, it doesn't mean the method is defect free or whether it's been entirely tested. For instance, if you've written a test for an if statement that contains a logical And and your tests exercise the left side of the expression, then a tool like Cobertura will report 100% line coverage, but in reality, you've only exercised 50% of the statement; therefore you've only achieved 50% branch coverage.

Now that you've written a bunch of tests, how do you determine what all those tests execute? Luckily, answering this question is where a code coverage tool like Cobertura shines. A code coverage tool reports test coverage -- either in the form of line or branch coverage -- that represents the amount of code that is being touched when a test is run.

Listing 8 demonstrates an Ant script that uses Cobertura to generate an HTML report of the code coverage achieved by running a series of JUnit tests:

Listing 8. Code coverage using Ant and Cobertura

<target name="instrument-classes"> <mkdir dir="${instrumented.dir}" /> <delete file="cobertura.ser" /> <cobertura-instrument todir="${instrumented.dir}"> <ignore regex="org.apache.log4j.*" /> <fileset dir="${classes.dir}"> <include name="**/*.class" /> <exclude name="**/*Test.class" /> </fileset> </cobertura-instrument> </target> <target name="run-instrumented-tests" depends="instrument-classes"> <mkdir dir="${logs.junit.dir}" /> <junit fork="yes" haltonfailure="true" dir="${basedir}" printsummary="yes"> <sysproperty key="net.sourceforge.cobertura.datafile" file="cobertura.ser" /> <classpath location="${instrumented.dir}" /> <classpath location="${classes.dir}" /> <classpath refid="test.class.path" /> <classpath refid="project.class.path"/> <formatter type="plain" usefile="true" /> <formatter type="xml" usefile="true" /> <batchtest fork="yes" todir="${logs.junit.dir}"> <fileset dir="${test.component.dir}"> <patternset refid="test.sources.pattern"/> </fileset> </batchtest> </junit> </target>

 

Cobertura produces an HTML report like the one in Figure 1. Notice the line and branch coverage percentages by package. You can click on each package to get class level line and path percentages and even see which source code lines were executed and how many times they was run.

Figure 1. HTML report produced using Cobertura and Ant

How much code coverage do you need?

Ideally, you want to have one test for each path. In other words, if your entire code base has a cyclomatic complexity of 20,000, you should have 20,000 tests. I've never run across a project with 100% path coverage, although I've seen teams with close to 100% line coverage.

You've seen various types of tests and even how to measure the coverage of those tests -- but how do you ensure that the execution of these tests happens at regular intervals? As it happens, this is where a CI server -- like CruiseControl steps up to the plate, as I'll show you next.

Running tests continually

Once you've incorporated the execution of these various developer test types into a build process, you can run some (or all) of these tests as part of a CI process. For example, in Listing 9, which is a snippet of CruiseControl's config.xml file, I define a number of things. First, I have CruiseControl monitoring a Subversion repository for any changes every two minutes. If it discovers any changes have occurred, CruiseControl kicks off a delegating build script (typically, you will see this written in Ant) called build-${project.name}.xml . The delegating build script calls the project's build script, which executes a compile and runs any tests.

I've also defined some logic to merge the results of all my different types of tests into a CruiseControl log file. Furthermore, I'm also using CruiseControl's capability to link (using the artifactspublisher tag) the reports generated by different tools into the Build Artifacts link, which is available from the CruiseControl dashboard application.

Listing 9. CI with CruiseControl

... <modificationset quietperiod="30"> <svn RepositoryLocation="http://your-domain.com/trunk/brewery" username="bfranklin" password="G0Fly@Kite"/> </modificationset> <schedule interval="120"> <ant anthome="apache-ant-1.6.5" buildfile="build-${project.name}.xml"/> </schedule> <log dir="logs/${project.name}"> <merge dir="projects/${project.name}/_reports/unit "/> <merge dir="projects/${project.name}/_reports/component "/> <merge dir="projects/${project.name}/_reports/performance "/> <merge dir="projects/${project.name}/_reports/functional "/> <merge dir="projects/${project.name}/_reports/coverage "/> </log> <publishers> <artifactspublisher dir="projects/${project.name}/_reports/" dest="projects/artifacts/${project.name}"/> </publishers> ...

 

You won't necessarily run every test defined with every source change applied to your version control repository. For instance, you can set up your CI system to execute a build that only runs unit tests during code check-ins (which is often referred to as a commit build). You can complement commit builds with more heavy-weight style builds that run component, functional, performance test and even code inspections, for example (Resources ). These builds can be run more infrequently (like once a day, for instance). Alternatively, you can run these tests and inspections immediately after your commit build.

---

Back to top

Calling all tests

Continuous testing encompasses both breadth and frequency. By authoring different types of tests, you can achieve a wider range of coverage and confidence; moreover, by running these tests continually, you can spot issues almost as fast as they are created.

Unit testing alone, at least as I've defined unit testing, won't get you too far on projects. Achieving higher code coverage and increasing a team's collective confidence takes a concerted effort to incorporate and execute automated component, performance, and functional tests as well. What's more, using frameworks and tools like JUnit, Selenium, and Cobertura enables easy build automation, which means with the help of a CI system, you can effectively execute your test suite every time a change is committed to your version control repository. That's a sure fire way to increase your batting average, don't you think?

 

Resources

Learn

• "Measure test coverage with Cobertura " (Elliotte Rusty Harold, IBM developerWorks, May 2005): Elliotte Rusty Harold shows you how to find untested code where bugs lurk via Cobertura.
  
• "Effective Unit Testing with DbUnit " (Andrew Glover, OnJava, January 2004): Andrew Glover introduces database-dependent testing with DbUnit.
  
• "In pursuit of code quality : Performance testing with JUnitPerf " (Andrew Glover, IBM developerWorks, November 2006): Two easy tests for monitoring scalability and performance.
  
• "Choosing a Continuous Integration Server " (Paul Duvall, IBM developerWorks, September 2006): A survey of open source CI servers: CruiseControl, Luntbuild, and Continuum.
  
• Use test categorization for agile builds (Andrew Glover, IBM developerWorks, October 2006): Andrew Glover reveals the three categories of testing needed to ensure end-to-end system soundness and then shows you how to automatically sort and run tests by category.
  
• Repeatable system tests (Andrew Glover, IBM developerWorks, September 2006): Andrew Glover introduces Cargo, an open source framework that automates container management in a generic fashion, so you can write logically repeatable system tests every time.
  
• "Automate acceptance tests with Selenium " (Christian Hellsten, IBM developerWorks, December 2005): How to use the Selenium test tool for functional testing of a Ruby on Rails and Ajax application.
  
• "Continuous Integration " (Martin Fowler): Fowler's seminal article on the practice of Continuous Integration.
  
• "JUnitPerf " (Mike Clark): Learn how to use JUnitPerf for early performance testing.
  
• Automation for the people series (Paul Duvall, IBM developerWorks): Explore the practical uses of automating software development processes and learn when and how to apply automation successfully.
  
• In pursuit of code quality (Andrew Glover, IBM developerWorks): Learn more about code metrics, test frameworks, and writing quality-focused code.
  
• The developerWorks Java technology zone : Hundreds of articles about every aspect of Java programming.
  

Get products and technologies

• JUnitPerf : Assess performance bottlenecks using JUnit.
  
• Cobertura : Calculate the percentage of source code accessed by tests.
  
• Selenium : A test tool for Web applications.
  
• DbUnit : A JUnit extension targeted for database-driven projects.
  
• CruiseControl : Framework for a continuous build process.