Automated Unit Testing

 

Introduction

Unit testing is an integral part of ensuring product quality, and it is the first chance for development teams to uncover and correct defects in their source code. When performed effectively, this form of testing provides significant gains in product quality, reduced development time/cost, source code design, and ease of maintenance.

Since unit testing is so poorly understood, a definition is provided here as a basis for further discussion. The term "module" in this definition represents a class in most object-oriented languages, and although this paper uses these terms interchangeably, a unit may not always be represented as such.

Chip-level testing for hardware is roughly equivalent to unit testing in software ?testing done on each module, in isolation, to verify its behavior. We can get a better feeling for how a module will react in the big wide world once we have tested it thoroughly under controlled (even contrived) conditions.

A software unit test is code that exercises a module. Typically, the unit test will establish some kind of artificial environment, then invoke routines in the module being tested. It then checks the results that are returned, either against known values or against the results from previous runs of the same test (regression testing). [Hunt]

According to Capers Jones, 15 ?50 percent of known defects are discovered during unit testing on average in the United States [Jones]. Defect-removal efficiency moves much higher when unit testing is combined with other quality assurance activities (e.g. inspections, reviews, integration testing, system testing, etc.). There is a direct correlation between a high defect removal efficiency and customer satisfaction.

Unfortunately, unit testing is often misunderstood and poorly implemented. This is largely a result of either not knowing how to perform unit testing effectively or not knowing that automation is possible. Other forms of testing currently suffer from the same misperceptions, so much so that testing in general is often seen as boring and unproductive. On the contrary, unit testing can be, dare I say it, fun and rewarding.

With all of the pressures surrounding software development teams, even the smallest improvement in efficiency and/or quality can mean a competitive edge. The goal of this paper is to examine the importance of unit testing, why developers resist it, and the benefits of automating the unit testing process.

Importance of Unit Testing

Before even discussing the benefits of automation, it is important to visit the importance of unit testing in general. The process of certifying individual modules working as intended is a first critical step in ensuring a quality implementation. This act alone provides a number of benefits, ranging from verification of programmer intent to ease of future maintenance.

Avoids Misunderstandings

Computers often do what we tell them to do instead of what we want them to do. This simple fact has caused almost every programmer considerable pain at one point or another. After all, the act of translating a design to a proper representation in an executable format is a highly error-prone process.

Unit testing is a valuable tool for ensuring that a module does what it is intended to do at two different levels. First, unit testing helps define and exercise a module抯 contract with the outside world. This means that the programmer will have some degree of confidence that the module does what it promises. Second, unit testing ensures that the internal implementation of the module performs as intended and properly supports its outside contract. These two forms of unit testing are commonly known as black box and white box testing respectively.

As an aside, the process of determining whether or not a given software module conforms to its design is a separate exercise that is not suitable for unit testing. This form of verification is best suited for reviews, walkthroughs, and inspections, which in their own right provide a number of benefits.

The bottom line: Unit testing gives programmers measurable confidence in the source code they produce.

Catches Problems Early

Unit testing gives you the opportunity to catch problems early while they are easy to detect. "Those tiny minnows have a nasty habit of becoming giant, man-eating sharks pretty fast, and catching a shark is quite a bit harder. [Hunt]" There are a number of statistics that show the cost benefit of catching defects as early as possible in the development life cycle. Consider the following data from two well respected companies:

IBM: An unpublished IBM rule of thumb for the relative costs to identify software defects: during design, 1.5; prior to coding, 1; during coding [unit testing falls here], 1.5; prior to test, 10; during test, 60; in field use, 100. [Humphrey]

TRW: The relative times to identify defects: during requirements, 1; during design, 3 to 6; during coding [unit testing falls here], 10; in development test, 15-40; in acceptance test, 30 to 70; during operation, 40 to 1000. [Humphrey]

As alluded to earlier, unit testing "catches problems long before any other testing method will reveal the defects. [Nygard]" The rules of thumb quoted above, as well as other research, confirms this commonly observed fact. Early detection of source code defects is the primary benefit of unit testing.

While a programmer is writing or making changes to source code, finding and correcting defects is usually trivial [McConnell] for a number of reasons. First, there is no "paperwork" involved in documenting the defect, because the source code has not been "checked-in" yet. Second, other programmers on the team do not rely on that source code, so there are no external dependencies. Third, defects are easier to find because they do not intermingle with defects in other modules.

The worst case scenario is of course for a customer to find a defect. With each defect that is found at this point, your relationship with that customer worsens, and the support/maintenance costs pile up quickly. Avoiding this scenario is virtually impossible; however, effective unit testing helps to lessen the frequency of such an occurrence to a much more manageable level.

The bottom line: Unit testing uncovers defects in source code shortly after it is written, which saves valuable time and resources, sometimes by orders of magnitude.

Focuses on the Interface

Unit testing is not simply a technique for finding defects in source code. It may also lead to simple, coherent implementation designs with well-defined interfaces. How is this possible? It is actually a side effect of writing unit tests before the implementation and then continuing incrementally between the two.

Writing unit tests first ensures that each module is tested effectively before integration. Most programmers write source code and then decide to test it, if at all. But if the testing of even one module is skipped, the testing becomes a burden, especially with poorly written code that is not designed to be tested in the first place. Software development teams often function under high-pressure situations, and the practice of writing tests first ensures that a programmer's short-term gain doesn抰 become a longer-term liability.

Writing unit tests first ensures that the module clearly conforms to its contract with the outside world and that the contract means what you think it does [Hunt]. This is especially important for programming languages that do not explicitly support design by contract, which is "the relationship between a class and its clients as a formal agreement, expressing each party抯 rights and obligations. [Meyer]" Simply put, the practice of writing tests first puts the programmer in the position of looking at the outside of the class (its interface), and ensuring that the interface accurately represents the function of the module in a clear and understandable manner. This has the positive affect of a module that is unambiguous in what it promises to do under certain conditions and that is just plain easier to use.

Writing unit tests first ensures that the requirements are properly represented in the module. In addition to representing these requirements verbatim as they are written, you also put yourself in a position to clarify them from the right perspective ?the interface, which is how clients of the module exercise its functionality. This serves as a catalyst for identifying exceptional conditions that are not specified in the requirements, specifically so that you take into account those things that may cause a module to fail.

Writing unit tests first forces you to write well-designed source code. After all, it is pretty much impossible to test spaghetti code. Testing first and by contract forces each operation in the module to have a clear and simple purpose, just as the module itself does. This effect even reaches into the implementation of the module, which is not observable from the outside, because these private operations must also be testable. This practice steers you clear of such common antipatterns as the blob, cut-and-paste programming, lava flow, poltergeists, and spaghetti code. [Brown] In case you are wondering, anitpatterns are not good.

The bottom line: Unit testing has the positive side effect of modules that are simple in purpose, that capture the appropriate requirements (and no more), that are easier to use, and that are easier to maintain.

Defines Completeness

How do you know when your module has been completed? Usually this is measured subjectively by programmers with varying degrees of success from person to person and from day to day. This is certainly not an appealing method for measuring completeness to a project manager who has to track progress. Unit testing is invaluable for this purpose.

As mentioned earlier, unit testing focuses on whether a module honors its contract. Once you test against this contract over a range of test cases and boundary conditions [Hunt], where the contract is likely to be broken, the module is effectively complete. An additional benefit here is a hedge against feature creep. One of the tenants of eXtreme programming [Beck] is keeping things as simple as possible, and testing against a module抯 contract helps to enforce this motto.

It should be noted that creating unit tests is not enough to define completion. Running all of them successfully is what defines completion. If anything less than 100 percent of all tests are running successfully, then you are not done.

You may be asking the following question at this point: How can I be done if some defects are bound to slip through? After all, you can't possibly think of every little thing that should be tested. The solution to this problem is explained later, but you simply add a test (less than 100 percent of tests run successfully) to uncover the defect and then fix it (exactly 100 percent of the tests now run successfully again). The end result is that each defect is found once and fixed once.

The bottom line: Unit testing helps you focus on exactly what is important for a module so that when all of your tests run successfully, you can be reasonably sure that your are done.

Reduces Debugging Time

The act of generating source code doesn't just involve writing it. In fact, a great deal of the average programmer抯 time is spent debugging the code that is written:

Debugging takes about 50 percent of the time spent in traditional, na飗e software-development cycle. Getting rid of debugging by preventing errors improves productivity. Therefore, the most obvious method of shortening a development schedule is to improve the quality of the product and decrease the amount of time spent debugging and reworking the software. [McConnell]

As discussed earlier, unit testing improves the design, usability, and maintainability of source code. This has a direct impact on reducing debugging time by actually reducing the number of defects that occur. In addition, defects that do still occur are easier to locate and fix, because the source code is well designed and documented with unit tests.

Unit testing catches defects in source code shortly after they are created. At this point they are easier to detect and fix simply because they are localized and not part of a cacophony of other defects at the same time. This directly reduces the amount of development time required for debugging.

The bottom line: Unit testing significantly reduces the amount of debugging necessary by avoiding defects in the first place and by catching those that do occur while they are relatively easy to detect and fix.

Avoids Big Bang Testing

Big bang testing is the common practice of not testing individual modules, otherwise known as incremental testing, until full system integration where everything usually blows up at once [Falk]. This has a number of disadvantages, all of which are fairly obvious and predictable.

With big bang testing, it is often very difficult to figure out what caused the system to fail and where the failure actually occurred. Since every module is likely to have defects, the water is significantly muddier than if each individual module had been tested apart from the system as a whole. Even worse, any number of these defects may be causing the failure. Testing at this point becomes much more difficult, if not practically impossible. Usually this is when long hours of debugging begin.

Big bang testing provides little or no testing automation. Once one defect is corrected, you must subjectively retest everything once more. Alternatively, unit tests and integration tests as a whole allow for quick regression testing through automation after a defect is addressed. Testing automation will be addressed later in this paper.

Big bang testing divides development teams by causing finger pointing. It is much easier to accuse someone else of causing the problem, when the actual cause of the problem itself is in doubt. With unit testing, there is no doubt who is responsible for the problem. And with a good suite of unit tests for each module, integration testing can actually focus on integration issues.

Big bang testing also encourages system testers to rush through their efforts, which usually occur at the last minute under this scenario. This leaves many of the defects to be discovered by your customers, which is very expensive and very bad for your reputation. Passing the buck to your customers because you didn't have the foresight to perform incremental testing is of little consolation at this point.

The bottom line: Unit testing avoids the practice of testing everything at once when it is significantly more expensive in terms of cost, development team morale, and customer satisfaction.

Gives Refactoring a Chance

Refactoring is the act of improving the internal design of source code without changing its external behavior [Fowler]. This is a common practice among more experienced programmers, and there are even automated tools to support common refactorings in some programming languages (Smalltalk being a good example of this). However, even these tools cannot automate every refactoring that you may wish to perform. This leads to the question of how to refactor safely without compromising existing code that works.

An essential precondition to refactoring is having a suite of solid unit tests [Fowler]. Imagine refactoring without tests. Once the changes are made, how do you know whether or not the module is still functioning properly? With unit tests, the fact that 100 percent of them still run successfully is a very good indication that all is well. Without this quick indicator, you must spend a great deal of time poking and prodding your module to get that warm, fuzzy feeling again.

Unit tests are a safety net for each module in your system. With automated unit testing, the benefits are even greater. As described in more detail later, all tests for a module can usually be run in seconds, giving you instant verification that it is still functioning properly. This also leads to quicker defect detection, because after each incremental change is made, the full suite of tests may be run quickly.

Without unit tests, refactoring is a dangerous prospect. Most programmers are very hesitant to change the design of source code that may work well enough. This is probably the biggest factor contributing to the steady decline in design coherence of most software systems over time. As new features are added and defects are addressed, the source code is not refactored to maintain a simple, coherent design. The source code eventually becomes a maintenance headache that most programmers would rather not even touch.

Unit tests prevent this problem. Not only do they act as a safety net for changes, they require that the source code remains testable. This really isn't as burdensome as it may sound; because as discussed earlier, better designed code is a side effect of thorough unit testing. I would even go as far as saying that unit testing actively encourages the act of refactoring, thereby extending the life and maintainability of a software system.

The bottom line: Unit testing serves as a safety net for refactoring. It also encourages the use of refactoring, thereby extending the life and maintainability of a software system.

Why Developers Resist

There are a number of reasons why programmers resist unit testing and even testing in general. These reasons should not be dismissed out of hand as grumbling, because there are often legitimate causes of this timidity. As a manager, forcing your programmers to unit test just because it is a requirement is not good enough. You will not end up with effective unit testing. A better way is showing them how unit testing makes them more productive and their life at the office less painful.

Lack of Time

This is perhaps the most common excuse for a quick and outright dismissal of unit testing. It is also very effective, because programmers in general are very busy, and adding unit testing would seem a burden to the uninformed observer (management). What the programmer is really doing here, though, is setting him and others up for a much more painful experience downstream.

You may counter that unit testing takes time away from programming. I would argue that unit testing is an integral part of programming itself. I say this because it is clear from an earlier discussion that unit testing actually improves the quality of source code. Now it is true that setting up a test harness and stubs for unit tests does take time. However, this relatively minor amount of time spent up front pays for itself in higher quality source code and less debugging. And of course, once the test harness is set up for one or more modules (it may be reused), adding additional unit tests is usually trivial. The time required to complete this staging activity is further reduced through the use of automated unit testing frameworks, which will be discussed in the next section.

In addition, automation provides a way to quickly exercise these unit tests over and over again in a matter of seconds or minutes at the most. This form of regression testing is of great benefit to any programmer that must make changes or correct defects in source code. Compare this to the long, arduous manual effort that must be repeated over and over again in the absence of an automated unit test suite. Suddenly unit testing doesn't look all that bad.

Most programmers are proud of their work and certainly want their peers to respect that same work. Without unit tests, you risk others uncovering your defects in large quantities. In addition, you don抰 benefit from the gains in quality and design clarity that unit tests provide as a side effect. That reasoning alone is worth consideration.

The bottom line: Unit testing actually saves time, not just by gains in quality and ease of maintenance, but through the use of automated regression testing.

Tests May Introduce Defects

You may argue that unit tests themselves may contain defects, which may actually retard programming efforts. This is certainly a valid concern, especially if unit tests are created with little thought and not packaged with the actual implementation source code for later reuse. There are some things you can do to prevent such a problem.

Treat unit tests just as you do the actual implementation. Construct them as carefully as the implementation, which includes careful inspection, stepping line-by-line in a debugger, walkthroughs and formal inspections, etc. Don't assume unit tests are somehow less error prone just because they are unit tests.

As noted earlier, debugging at the unit level is much easier than after integration occurs. This is important, because unit tests don't usually have their own unit tests. In addition to the careful steps taken to ensure their quality, debugging unit tests is fairly straightforward when necessary because each test should be simple in nature and specific in its intent.

Treat unit tests as if they are an integral part of the actual implementation source code. This will ensure that unit-testing code will be of the same high standard and less likely to be overridden with time-consuming defects. In practice, placing unit tests under revision control, actually packaging them with the module they actually test, and requiring that they be run whenever changes are made to the corresponding module effectively meets this goal.

The bottom line: The quality of unit tests does not become an issue if they are packaged with and held to the same standards as the actual implementation.

Lack of Humility

Most programmers have been humbled over time by the complexity of their work. However, there are some out there who insist that the source code they produce is next to perfect in terms of quality without the need for such things as unit testing. Although it is certainly true that some programmers are more proficient than others [McConnell], such an assertion is ridiculous, especially when you consider that most development efforts involve teams of programmers.

A simple fact worth noting here is that we are all human (as far as I know). Humans make mistakes, especially when trying to express requirements and designs formulated mostly in English to machine digestible code. All programmers will generate defects.

In addition, the source code that most programmers produce is relied upon by others. Ethically speaking, this means each programmer has a certain duty to ensure that each module he produces functions properly. Others may argue with me here, but I do not think this requirement is unreasonable under most circumstances.

Furthermore, how do you show with some degree of certainty that a certain module works as advertised? You certainly cannot just say that it does. Most people will reject this as insufficient if they have a choice. You also cannot simply show that it works by running it a few times. Although this display is more comforting, it does leave some nagging doubts. On the other hand, a full suite of unit tests that all run successfully is much more reassuring.

The bottom line: Saying your code works isn抰 worth much. Showing that your code works with unit tests is valuable.

Legacy Code

There is a fantastically large amount of source code out there that must be maintained. Much of this code is also showing its age. What do you do when faced with thousands of lines of code that have no existing unit tests? It is very tempting to dismiss the idea of unit testing at this point as a pointless exercise in futility. At the same time, even breathing on aged source code is a frightening thought. What if your one little change lights a chain reaction of errors that leads to a total melt down? There actually is a way to use unit tests, even on existing source code.

If you need to make a modification, then create the necessary test fixture (or scaffolding) around it. Then create tests that verify the piece of code you are modifying. At this point, you may begin the process of making a modification, usually in deliberate, steady steps. After each step, run the unit tests to verify that everything works okay. If there is a problem, you know exactly what caused it, and you can easily take a step back. Since you have scaffolding and working tests to support the existing structure, you don't have to worry about the walls coming down while you rebuild them [Jeffries].

If you need to fix a defect, then follow a similar path. The only exception here is that each test you write should expose the defect by failing. Once each of these tests runs successfully, then the defect has been addressed. However, this is not the end of the story. Package the fixture and tests with the modified source code, so that the next time a defect or modification must be addressed, you already have a fixture and test suite that can be built upon. Next time it will be easier.

The biggest mistake most programmers make is thinking they must fix everything at once. Taking changes one step at a time makes the process of maintenance a much more manageable task. And unit tests serve as a safety precaution against a collapse while you work. Later on they will serve as an insurance policy against changes that may undo your existing work.

The bottom line: Unit tests serve as an important safety measure while you work in maintenance mode and an insurance policy later on when additional changes are made.

What to Test

The question of how to test will be examined in the next section. Before going down that road, it is important to know what to test for in the first place. After all, what good is a bunch of unit tests if they don't test for the right things. Not knowing what to test for is a large impediment to writing unit tests for the first time, so thankfully there are a number of common things to look for [McConnell]:

• Each requirement that applies to the module in question should have one or more unit tests that ensure it is properly addressed.
• Each design element, if a documented design exists, that applies to the module in question should have one or more unit tests that ensure it is properly addressed.
• Common error conditions that you have experienced in the past or that have appeared in other modules is always a good place to look.
• Each data-flow condition is ripe for unit tests. Essentially, each conditional (if-then, while, etc.) in a module抯 operations could have a test for a true condition and a false condition.
• Boundary conditions are an important target for unit tests. What you ideally want to do is test with values just below, at, and above the boundary and compare the results you expect with the ones you get.
• Compound boundaries are often overlooked. You may wish to check the results of a multiplication or division of two variables to make sure they aren't too large. You may also wish to check the effect of assigning large strings to variables that may expect a much smaller length.
• Checking for bad data is a relatively easy class of unit tests. This includes things such as incomplete data, too much data, invalid data, the wrong size of data, or uninitialized data.
• If a module must support backward compatibility with older data or business operations, then each of these cases should be verified by one or more unit tests.

The above list is by no means exhaustive, but it does provide some direction for where to start. And if while adding one test you happen to think of others, implement them while they are fresh in your mind. Too many tests aren't necessarily a bad thing, unless they prevent you from starting in the first place. You can always go back later and remove or consolidate tests as necessary.

When do you stop writing tests? What you should always do first is focus on those parts of a module that are the most complex or the most likely to produce errors. Trying to test everything under the sun and every possible path is not practical. Focus on where the risk is [Fowler], and when you are comfortable that all of your worries have been addressed, then you are probably done. Also keep in mind that unit testing is itself an incremental approach to quality. If you think of something later, add a test. Likewise, if a defect slips through, write a test to uncover it and then fix it.

The bottom line: When writing unit tests, focus on where the risk is.

Bugs Still Slip Through

No form of testing will discover all defects that are present. This is just a simple fact that must be understood and accepted. The good news is that each defect that is detected early on by unit testing could potentially save a significant amount of time and money, not to mention your organization's good reputation with its customers.

If that doesn't convince you, then think of the long hours of debugging you could be saving yourself and your colleagues. Wouldn't you rather find most of your defects before you release your source code to others? Finally, wouldn't you rather spend more time designing and implementing your next masterpiece instead of trudging around looking for one elusive defect among many?

There are clearly benefits to be gained from unit testing, even if some defects slip through. And since unit testing is an incremental process, you will certainly write a test to expose any defect that slips through so that it never occurs again. Finding defects downstream is almost expected. Finding the same defect more than once is almost universally frowned upon.

The bottom line: Unit tests won抰 uncover all defects at once, but they will ensure that no single defect occurs more than once.

Nobody Else Does It

This is a particularly dangerous excuse that keeps the practice of unit testing from ever taking hold. In fact, this excuse is a major cause of failing to implement anything that is new to an organization. It also ignores why nobody else has thought of or implemented unit testing.

• Perhaps someone else did introduce this idea but was ineffective in their effort to show its benefits. In order to introduce this practice, you must show programmers and management that there is a clear benefit to them.
• Perhaps someone who introduced this idea was flatly rejected regardless of the benefits. If you cannot work around this individual, then perhaps there are greener pastures?/LI>
• Perhaps the idea of unit testing has never been presented to your group. Or perhaps a less beneficial approach was presented and rejected.
• Perhaps the people in your group or organization simply lack the necessary information to make efficient use of unit testing. Don't miss an opportunity to present such valuable information when people are ready to receive it. Showing initiative isn't a crime, but withholding valuable information from people who are willing to accept it might as well be a crime.

There is no comfort in going along with the crowd when you are routinely faced with long hours of debugging. Take the initiative and show others that there is a better way.

The bottom line: Don't discard the idea of unit testing just because nobody else is doing it. You may be missing an opportunity to make life easier.

Benefits of Automation

Now that we've discussed what the benefits of unit testing are and even what things we may want to test for, it's time to see what one of these unit tests actually looks like. If you are like me, you probably learn best by example, so the discussion in this section will be based on an example that is provided in its entirety in Appendix A (implementation) and B (unit tests).

NOTE: The example used in this article is probably out of date, so make sure you check with the JUnit site before starting your own unit testing.

There are a couple of things you may notice about this example. First, the programming language used is Java. You don't have to understand Java, but you should understand one or more programming languages in general. Second, the suite of unit tests in Appendix B depends on other classes not shown here. These other classes are an automated unit-testing framework for Java called JUnit [JUnit]. It makes the job of creating, organizing, and running unit tests much easier and faster. Incidentally, this particular framework is available in a number of other programming languages and development environments.

Frameworks like this are very important when, as is often the case, you write a large number of unit tests. They are equally important when using the good practice of writing unit tests and the module's they test in tandem. This is because you will want to run all of the tests for a module (and maybe others as well) very often. Doing this by hand is tedious work. Frameworks allow us to realize the benefits of unit testing in an efficient manner.

Even if you decide not to use JUnit or one of its siblings, the one you use should offer the following features [Hunt]:

• A standard way to specify setup and cleanup
• A simple method for organizing tests
• A simple method for choosing which tests to run
• A simple method for comparing actual results to expected results
• A standard form of error and failure reporting

I won't spend time explaining every line of code in the example, because this is done adequately by the documentation provided with JUnit [JUnit]. My goal in this section is to show you what a suite of unit tests looks like and how a framework automates common unit testing tasks.

Standard Setup and Cleanup

As discussed earlier, each module should be tested in isolation from the rest of the system. Our example module is a Java class. Exercising this class in isolation allows us to test mercilessly without having to worry about the rest of the software system it is contained within.

Before we continue, notice how the MoneyTest class subclasses the TestCase class. This is done so that JUnit recognizes MoneyTest as a unit test or suite of unit tests, thereby allowing us to "hook in" to the JUnit framework.

Notice the following snippet of source code from MoneyTest:

    protected void setUp() {
        m12CHF = new Money(12, "CHF");
        m14CHF = new Money(14, "CHF");
    }

Before JUnit executes a unit test, it first calls the setUp method. This method may be used to construct the necessary test fixture. In this example we are merely creating two Money objects that will be used in all of the unit tests in the MoneyTest suite. Often this method is much meatier, depending on what conditions you must simulate for your tests. There is a matching method called tearDown that isn't necessary for this example. You would use this method to clean up after each unit test that is run, such as closing files, closing database connections, removing temporary files, etc.

When JUnit executes the MoneyTest suite, the following operations are performed in order:

• Run setUp, run testMoneyEquals, and run tearDown
• Then run setUp, run testSimpleAdd, and run tearDown

As you can see, the test fixture is setup and torn down independently for each unit test in this suite. This ensures that each test is run in isolation. What if you do not implement the setUp and/or tearDown methods in your suite? JUnit will still run these methods, which are defined with empty bodies further up the inheritance heirarchy in the framework, but the net effect is that only your unit tests will run.

Test Organization

Test suites in JUnit are composable. In other words, each suite may contain any number of tests and/or suites. Let's take a look at another snippet of source code from MoneyTest:

    public static Test suite() {
        TestSuite suite = new TestSuite();
        suite.addTest(new MoneyTest("testMoneyEquals"));
        suite.addTest(new MoneyTest("testSimpleAdd"));
        return suite;
    }

When JUnit executes the MoneyTest suite, it basically runs the suite method. This is another framework method, similar to setUp and tearDown, that allows you to register unit tests and other suites. Each of these registered tests and test suites would make up the whole of MoneyTest. This simple registration method allows us to build large hierarchies of tests that can be executed from a single point.

Let's take a look at one more snippet of code from MoneyTest:

    public void testSimpleAdd() {
        Money expected = new Money(26, "CHF");
        Money result = m12CHF.add(m14CHF);

        assert(expected.equals(result));
    }

This is a unit test. A unit test in JUnit is simply a method in a suite that begins with the word test. When JUnit executes a suite, it runs each internally registered test and suite.

Selection of Tests

If the framework you are using supports composable test suites, then you should be able to choose your execution point at any place in the suite hierarchy. This allows you to run one, some, or all tests at any given time. JUnit gives you a command line (or batch) interface and a graphical interface for running tests.

An example of the JUnit graphical interface is provided below:

This example shows a hierarchy of suites, one of which is selected and run. You may visually select any node in the tree as your execution point, whether it's an individual test or a suite.

An example of the JUnit command line interface is given below:

    java junit.textui.TestRunner MoneyTest

This simple command runs all registered tests and suites in the MoneyTest suite. For larger suites of tests, you move the execution point by simply changing the name of the suite in the above command. This form of test execution is very useful while programming and for ensuring that all tests run in build makefiles.

Comparing Actual to Expected Results

The framework you use should provide a simple method for comparing actual results to expected results. JUnit does this with the use of assertions. Consider the following unit test from our example:

    public void testMoneyEquals() {
        assert(!m12CHF.equals(null));
        assertEquals(m12CHF, m12CHF);
        assertEquals(m12CHF, new Money(12, "CHF"));
        assert(!m12CHF.equals(m14CHF));
    }

You gain access to the assert method by subclassing TestCase. This method evaluates the argument for a true or false value. The first use of this method is to determine whether or not the m12CHF Money object is null. If it is null, then the assertion fails and this test fails. If it is not null, then the assertion succeeds and this test continues. The assertEquals method is simply a convenience variant of assert that compares two objects for equality. JUnit offers an array of these variants to make life easier.

These assertions are where the real action occurs in a JUnit unit test. This is where you compare those actual results to expected results.

Standard Error and Failure Reporting

The testing framework you use should provide a standard means of reporting errors and failures. When an assertion fails, that is a failure. When something unexpected happens, like an exception, that is an error. It is important to have a summary of what actually ran, how long it took to run, and whether or not all tests completed successfully. If something did go wrong, you should at a minimum see what test failed. The end result of such reporting is very little wasted time trying to find out what happened.

You've already seen what the output from a successful JUnit run looks like from the graphic above. Here is what the command line interface reports to us with our example:

    ..
    Time: 0.05
    OK (2 tests)

Each dot stands for a test that was run. The time indicates how long it took to run the unit tests. The OK means that two tests ran successfully. This is really all we need to know when the runs are successful.

Now let's force one of our tests fail. Take, for example, the first assertion in testMoneyEquals:

    assert(!m12CHF.equals(null));

If we remove the negation, then we are asserting that m12CHF is null.

    assert(m12CHF.equals(null));

This is clearly not the case, because an object is being assigned to this variable before the test is run in the setUp method. The new JUnit output should look like this:

    .F.
    Time: 0.11

    FAILURES!!!
    Test Results:
    Run: 2 Failures: 1 Errors: 0
    There was 1 failure:
    1) testMoneyEquals(com.acme.money.MoneyTest)
    junit.framework.AssertionFailedError
        at junit.framework.Assert.fail(Assert.java:143)
        at junit.framework.Assert.assert(Assert.java:19)
        at junit.framework.Assert.assert(Assert.java:26)
        at com.acme.money.MoneyTest.testMoneyEquals(MoneyTest.java:40)
        at java.lang.reflect.Method.invoke(Native Method)
        at junit.framework.TestCase.runTest(TestCase.java:155)
        at junit.framework.TestCase.runBare(TestCase.java:129)
        at junit.framework.TestResult$1.protect(TestResult.java:100)
        at junit.framework.TestResult.runProtected(TestResult.java:117)
        at junit.framework.TestResult.run(TestResult.java:103)
        at junit.framework.TestCase.run(TestCase.java:120)
        at junit.framework.TestSuite.run(TestSuite.java:144)
        at junit.textui.TestRunner.doRun(TestRunner.java:61)
        at junit.textui.TestRunner.run(TestRunner.java:181)
        at junit.textui.TestRunner.run(TestRunner.java:167)
        at com.acme.money.MoneyTest.main(MoneyTest.java:31)

JUnit reports that testSimpleAdd in the MoneyTest suite failed. We are also given a detailed stack trace that shows us the failure occurred on line 40 of MoneyTest. As expected, this line contains the assertion we forced to fail. You can see how this kind of reporting makes the job of uncovering and locating errors and failures easy.

Regression Testing

The speed and accuracy that automation gives us with unit testing makes most regression testing a piece of cake. Consider the following possibilities:

• Since no babysitting is required, you may run automated tests as often as you like. Usually each run takes only seconds.
• Automated tests are great bug detectors, because you may run them after each compile. Instant feedback is provided on whether or not your change broke something. Since the new source code is fresh in your mind, very little debugging, if any, is required.
• As a manager or technical lead, you may now require each programmer to run all unit tests (even from other programmers) for the product before checking in (or integrating) new source code. This is possible because the tests will run quickly and check their own results. If a programmer sees that OK in the summary output, then they are cleared for landing. If any of the tests fail, then that programmer is at fault and must have all tests running successfully before integration.

Regression testing would be much more difficult without automation, so this is probably the biggest benefit that automation provides.

Conclusion

Unit testing provides significant gains in product quality, reduced development time/cost, source code design, and ease of maintenance. Automated unit testing enhances those gains and gives you fast and accurate regression testing as well. I would encourage you to give JUnit or other unit testing frameworks like it more than just a passing glance. You may actually find testing fun and productive.

I now leave you with some wise words of wisdom from The Pragmatic Programmer [Hunt]:

"Test Your Software, or Your Users Will"

"Test Early. Test Often. Test Automatically"

"Coding Ain't Done 'Til All the Tests Run'"

"Find Bugs Once"

Happy testing!

References

[Beck]
Beck, Kent. Extreme Programming Explained: Embrace Change. Reading, MA: Addison-Wesley, 1999.

[Beck Test]
Beck, Kent and Erich Gamma. Test Infected: Programmers Love Writing Tests. http://www%20junit.org/.

[Brown]
Brown, William J., Raphael C. Malveau, Hays W. McCormick III, and Thomas J. Mowbray. AntiPatterns: Refactoring Software, Architectures, and Projects in Crisis. New York, NY: Wiley Computer Publishing, 1998.

[Fowler]
Fowler, Martin, Kent Beck, John Brant, William Opdyke, and Don Roberts. Refactoring: Improving the Design of Existing Code. Reading, MA: Addison-Wesley, 1999.

[GoF]
Gamma, Erich, Richard Helm, Ralph Johnson, and John Vlissides. Design Patterns: Elements of Reusable Object Oriented Software. Reading, MA: Addison-Wesley, 1995.

[Humphrey]
Humphrey, Watts S. A Discipline for Software Engineering. Reading, MA: Addison-Wesley, 1995.

[Hunt]
Hunt, Andrew and David Thomas. The Pragmatic Programmer. Reading, MA: Addison-Wesley, 1999.

[Jeffries]
Jeffries, Ron, Ann Anderson, Chet Hendrickson, and Ronald Jeffries. Extreme Programming Installed. Reading, MA: Addison-Wesley, 2000.

[Jones]
Jones, Capers. Software defect-removal efficiency. IEEE Computer. Volume 29, Number 2. Pages 94-95, April 1996.

[JUnit]
Beck, Kent and Erich Gamma. JUnit Open-Source Testing Framework. Available on the web at http://www.junit.org/.

[Kaner]
Kaner, Cem, Jack Falk, and Hung Quoc Nguyen. Testing Computer Software. Second Edition. New York, NY: Wiley Computer Publishing, 1999.

[McConnell]
McConnell, Steve. Code Complete: A Practical Handbook of Software Construction. Redmond, WA: Microsoft Press, 1993.

[Meyer]
Meyer, Bertrand. Object Oriented Software Construction. Second Edition. Upper Saddle River, NJ: Prentice Hall, 1997.

[Nygard]
Nygard, Michael and Tracie Karsjens. Test Infect Your Enterprise Beans. JavaWorld. 2000.

Appendix A

The following example class is found in an article that is provided with the JUnit framework to illustrate its use [Beck Test]. It represents money, which may have an amount and currency. You may also add one money object to another and compare two money objects for equality. The unit tests for this class are provided in Appendix B.

package com.acme.money;

/**
 * Represents a chunk of money with an amount and currency.
 *
 * @author  Kent Beck
 * @author  Erich Gamma
 * @author  Bill Willis (modifications)
 *
 * @version 1.0
 */

public class Money {
    private int fAmount;
    private String fCurrency;

    public Money(int amount, String currency) {
        fAmount = amount;
        fCurrency = currency;
    }

    public int amount() {
        return fAmount;
    }

    public String currency() {
        return fCurrency;
    }

    public Money add(Money m) {
        return new Money(amount() + m.amount(), currency());
    }

    public boolean equals(Object anObject) {
        if (! anObject instanceof Money)
            return false;

        Money aMoney = (Money)anObject;
        return aMoney.currency().equals(currency())
            && amount() == aMoney.amount();
    }
}

Appendix B

The following example class is found in an article that is provided with the JUnit framework to illustrate its use [Beck Test]. It is a test suite that contains some unit tests for the class provided in Appendix B.

package com.acme.money;

import junit.framework.*;

/**
 * This test suite contains a number of unit tests that verify
 * Money.
 *
 * @author  Kent Beck
 * @author  Erich Gamma
 * @author  Bill Willis (modifications)
 *
 * @version 1.0
 */

public class MoneyTest extends TestCase {

    private Money m12CHF;
    private Money m14CHF;

    public static Test suite() {
        TestSuite suite = new TestSuite();
        suite.addTest(new MoneyTest("testMoneyEquals"));
        suite.addTest(new MoneyTest("testSimpleAdd"));
        return suite;
    }

    public MoneyTest(String name) {
        super(name);
    }

    public static void main(String args[]) {
        junit.textui.TestRunner.run(MoneyTest.class);
    }

    protected void setUp() {
        m12CHF = new Money(12, "CHF");
        m14CHF = new Money(14, "CHF");
    }

    public void testMoneyEquals() {
        assert(!m12CHF.equals(null));
        assertEquals(m12CHF, m12CHF);
        assertEquals(m12CHF, new Money(12, "CHF"));
        assert(!m12CHF.equals(m14CHF));
    }


    public void testSimpleAdd() {
        Money expected = new Money(26, "CHF");
        Money result = m12CHF.add(m14CHF);

        assert(expected.equals(result));
    }
}

About the Author

Bill Willis is the director of engineering for ObjectVenture Inc., where he spends a great deal of time developing new pattern-based tools and technology. He is also the director of PatternsCentral, an online community and portal devoted to helping people make more effective use of patterns.