The art of software testing

Black-box testing:

To use this method, view the program as a black box.Your goal is to be completely unconcerned about the internal beharvior and structure of the program. Instead, concentrate on finding circumstance in which the program does not behave according to its specifications.

     In this approach, test data are derived solely from thespecifications.

     If you want to use this approach to findall errors in the program, the criterion is exhaustive input testing, makinguse of every possible input condition as a test case. Why? If you tried threeequilateral triangle test cases for the triangle program, that in no way guaranteesthe correct detection of all the equilateral triangles. The program couldcontain a special check for values 3842, 3842, 3842 and denote such a triangleas a scalene triangle. Since the program is a black box, the only way to besure of detecting the presence of such a statement is by trying every inputcondition.

     To test the triangle program exhaustively, you would have tocreate test cases for all valid triangles up to maximum integer size ofdevelopment language. This in itself is an astronomical number of test cases, butit is in no way exhaustive.it would find errors where the program said that -3,4, 5 is a scalene triangle and that 2, A, 2 is an isosceles triangle, to besure of finding all such errors, you have to test using not only all validinputs, but all possible inputs. Hence, to test triangle program exhaustively,you would have to produce virtually an infinite number of test cases, which, ofcourse, is not possible.

     If this sounds difficult, exhaustive input testing of larger programs is even more of aproblem. Consider attempting an exhaustive black-box test of a C++ compiler.Not only would you have to create test cases representing all valid C++programs, but you would have to create test cases for all invalid C++ programsto ensure that the compiler detects them as being invalid. That is, the compiler has to be tested to ensure that it does not do what it is not supposed to do-for example, successfully compile a syntactically incorrect program.

    The problem is even worse for programs having a memory, suchas operating systems or database applications. For example, in a databaseapplication such as an airline reservation system, the execution of transactionis dependent upon what happened in previous transactions. Hence, not only wouldyou have to try all unique valid and invalid transactions, but also allpossible sequences of transactions.

    This discussionshows that exhaustive input testing is impossible. Two implications of this arethat (1) you cannot test a program to guarantee that it is error free and that (2)a fundamental consideration in program testing is one of economics. That is,since exhaustive testing is out of the question, the objective should be to maximizethe yield on testing investment by maximizing the number of error found by afinite number of test cases. Doing so will involve, among other things, beingable to peer inside the program and making certain reasonable, but notairtight, assumptions about the program, for example, if the triangle programdetects 2, 2, 2 as an equilateral triangle, it seems reasonable that it will dothe same for 3, 3, 3. This will form part of the test-case-design strategy inchapter 4.

White-Box Testing:

Another testing strategy, white-box or logic-driven testing, permits you to examine the internal structure of the program. This strategy derives test data from an examination of the program’s logic (and often, unfortunately, at the neglect of the specification).

     The goal at this point is to establish, for this strategy, the analog to exhaustive input testing in the black-box approach. Causing every statement in the program to execute at least once might appear to be the answer, but it is not difficult to show that this is highly inadequate. Without belaboring the point, since this matter is discussed in more depth in Chapter 4, the analog is usually considered to be exhaustive path testing. That is, if you execute, via test cases, all possible paths of control flow through the program, then possibly the program has been completely tested.

    There are two flaws in this statement, however. One is that the number of unique logic paths through a program could be astronomically large. To see this, consider the trivial program represented in Figure 2.1. The diagram is a control-flow graph. Each node or circlerepresents a segment of statements that execute sequentially, possibly terminating with a branching statement. Each edge or arc represents a transfer of control (branch) between segments. The diagram, then, depicts a 10- to 20-statement program consisting of a DO loop that iterates up to 20 times. Within the body of the DO loop is a set of nested IF statements. Determining the number of unique logic paths is the same as determining the total number of unique ways of moving from point a to point b (assuming that all decisions in the program are independent from one another). This number is approx-imately 1014, or 100 trillion. It is computed from 520 + 519 + . . . 51, where 5 is the number of paths through the loop body. Since most people have a difficult time visualizing such a number, consider it this way: If you could write, execute, and verify a test case every five minutes, it would take approximately one billion years to try every path. If you were 300 times faster, completing a test once per second, you could complete the job in 3.2 million years, give or take a few leap years and centuries.

     Of course, in actual programs every decision is not independent from every other decision, meaning that the number of possible execution paths would be somewhat less. On the other hand, actual programs are much larger than the simple program depicted in Figure 2.1. Hence, exhaustive path testing, like exhaustive input testing, appears to be impractical, if not impossible. The second flaw in the statement “exhaustive path testing means a complete test” is that every path in a program could be tested, yet the program might still be loaded with errors. There are three explanations for this.

    The first is that an exhaustive path test in no way guarantees that a program matches its specification. For example, if you were asked to write an ascending-order sorting routine but mistakenly produced adescending-order sorting routine, exhaustive path testing would be of little value; the program still has one bug: It is the wrong program, as it does not meet the specification.

    Second, a program may be incorrect because of  missing paths. Exhaustive path testing, of course, would not detect the absence of necessary paths.

    Third, an exhaustive path test might not uncover data-sensitivity errors. There are many examples of such errors, but a simple example should suffice. Suppose that in a program you have to compare two numbers for convergence, that is, to see if the difference between the two numbers is less than some predetermined value. For example, you might write a Java IF statement as

if (a-b < c)

System.out.println("a-b < c");

    Of course, the statement contains an error because it should compare c to the absolute value of a-b. Detection of this error, however, is dependent upon the values used for a and b and would not necessarily be detected by just executing every path through the program. In conclusion, although exhaustive input testing is superior to exhaustive path testing, neither proves to be useful because both are infeasible. Perhaps, then, there are ways of combining elements of black-box and white-box testing to derive a reasonable, but not airtight, testing strategy. This matter is pursued further in Chapter 4.