Chp 2 The Psychology and Economics of Program Testing
The Psychology of Testing
Testing is the process of executing a program with the intent of finding errors.
The most important considerations in software testing are issues of psychology,
Human beings tend to be highly goal-oriented, and establishing the proper goal has an important psychological effect.
Testing is a destructive process, even a sadistic process, which explains why most people find it difficult.
Atest case that finds a new error can hardly be consideredunsuccessful;rather, it has proven to be a valuable investment. Anunsuccessful test case is one that causes a program to produce thecorrect result without finding any errors.
Psychological studies tell us that people perform poorly when they setout on a task that they know to be infeasible or impossible.
Black-Box Testing(data-driven testing)
exhaustive input testing is impossible
White-Box Testing(logic-driven tesing)
exhaustive path testing, like exhaustive input testing, appears to be impractical, if not impossible.
Software Testing Principles
Principle 1: A necessary part of a test case is a definition of the expected output or result.
Principle 2: A programmer should avoid attempting to test his or her own program.
Principle 3: A programming organization should not test its own programs.
Principle 4:Thoroughly inspect the results of each test.
Principle 5:Test cases must be written for input conditions that are invalid and unexpected, as well as for those
that are valid and expected.
Principle 6: Examining a program to see if it does not do what it issupposed to do is only half the battle; the other half is seeingwhether the program does what it is not supposed to do.
Principle 7: Avoid throwaway test cases unless the program is truly athrowaway program.Saving test cases and running them again afterchanges to other components of the program is known as regression testing.
Principle 9:The probability of the existence of more errors in a section of a program is proportional to the number
of errors already found in that section.
Principle 10:Testing is an extremely creative and intellectuallychallenging task.It is probably true that the creativity required intesting a large program exceeds the creativity required in designing that program.
Chp 3 Program Inspections, Walkthroughs, and Reviews
nspections/walkthroughs and computer-based testing are complementary
The simpleact of reading aloud a program to an audience seems to be a remarkably effective error-detection technique.
For the inspection process to be effective, the appropriate attitude must be established.
Chp 4 Test-Case Design
Test-case design is so important because complete testing is impossible.
Chapter 2 showed that exhaustive black-box and white-box testing are,in general, impossible, but suggested that a reasonable testingstrategy might be elements of both.
The recommended procedure is to develop test cases using the black-box methods and then develop supplementary test cases as necessary with white-box methods.
Logic-Coverage Testing
If you back completely away from path testing, it may seem that aworthy goal would be to execute every statement in the program at leastonce. Unfortunately, this is a weak criterion for a reasonable white-box test.
A stronger logic-coverage criterion is known as decision coverage or branch coverage.
Decision coverage usually can satisfy statement coverage.Decisioncoverage is a stronger criterion than statement coverage, but it stillis rather weak. A criterion that is sometimes stronger than decisioncoverage is condition coverage.
condition coverage usually is superior to decision coverage.Althoughthe condition-coverage criterion appears, at first glance, to satisfythe decision-coverage criterion, it does not always do so.
The obvious way out of this dilemma is a criterion called decision/condition coverage. It requires sufficient test cases that each condition in a decision takes on all possible outcomes at least once, each decision takes on all possible outcomes at least once, and each point of entry is invoked at least once.
Equivalence Partitioning
you need to select the right subset, the subset with the highestprobability of finding the most errors. One way of locating this subsetis to realize that a well-selected test case also should have two other properties:
1.It reduces, by more than a count of one, the number of other testcases that must be developed to achieve some predefined goal of“reasonable” testing.
2.It covers a large set ofother possible test cases. That is, it tells us something about thepresence or absence of errors over and above this specific set of inputvalues
These two properties, although they appear to be similar, describe two distinct considerations.These two considerations form a black-box methodologyknown as equivalence partitioning.The second consideration is used todevelop a set of “interesting” conditions to be tested. The firstconsideration is then used to develop a minimal set of test casescovering these conditions.
Test-case design by equivalence partitioning proceeds in two steps:
(1) identifying the equivalence classes.
Given an input or external condition, identifying the equivalence classes is largely a heuristic process.
(2) defining the test cases.
The second step is the use of equivalence classes to identify the test cases. The process is as follows:
1. Assign a unique number to each equivalence class.
2. Until all valid equivalence classes have been covered by (incorporated into) test cases, write a new test case covering as many of the uncovered valid equivalence classes as possible.
3. Until your test cases have covered all invalid equivalence classes, write a test case that covers one, and only one, of the uncovered invalid equivalence classes.
Although equivalence partitioning is vastly superior to a randomselection of test cases, it still has deficiencies. It overlookscertain types of high-yield test cases, for example.
Boundary-Value Analysis
Experience shows that test cases that explore boundary conditions have a higher payoff than test cases that do not.
Boundary-value analysis differs from equivalence partitioning in two respects:
1.Rather than selecting any element in an equivalence class asbeingrepresentative, boundary-value analysis requires that one or moreelements be selected such that each edge of the equivalence class isthe subject of a test.
2. Rather than just focusingattention on the input conditions (input space), test cases are alsoderived by considering the result space (output equivalence classes).
Boundary-value analysis, if practiced correctly, isone of the mostuseful test-case-design methods. However, it often is usedineffectively because the technique, on the surface, sounds simple.
Cause-Effect Graphing
One weakness of boundary-value analysis and equivalence partitioning is that they do not explore combinations of input circumstances.
The testing of input combinations is not a simple task because even ifyou equivalence-partition the input conditions, the number ofcombinations usually is astronomical.
Cause-effect graphing aids in selecting, in a systematic way, ahighyield set of test cases. It has a beneficial side effect inpointing out incompleteness and ambiguities in the specification.
Cause-effect graphing requires the translation of a specification intoa Boolean logic network, it gives you a different perspective on, andadditional insight into, the specification. In fact, the development ofa cause-effect graph is a good way to uncover ambiguities andincompleteness in specifications.
Themost difficult aspect of the technique is the conversion of the graphinto the decision table. This process is algorithmic, implying that youcould automate it by writing a program
The Strategy
Thetest-case-design methodologies discussed in this chapter can becombined into an overall strategy. The reason for combining them shouldbe obvious by now: Each contributes a particular set of useful testcases, but none of them by itself contributes a thorough set of testcases.
Chp 5 Module (Unit) Testing
Test-Case Design
You need two types of information when designing test cases for a module test: a specification for the module and the module’s source code
Module testing is largely white-box oriented.
the multicondition-criterionis superior to the other criteria, and any logic-coverage criterion isnot good enough to serve as the only means of deriving module tests.
Incremental Testing
Thequestion pondered here is the following: Should you test a program bytesting each module independently and then combining the modules toform the program, or should you combine the next module to be testedwith the set of previously tested modules before it is tested
The testing of each module requires a special driver module and one or more stub modules.
1.Nonincremental testing requires more work.
2.Programming errors related to mismatching interfaces or incorrectassumptions among modules will be detected earlier if incrementaltesting is used.
3.debugging should be easier if incremental testing is used.
4.Incremental testing might result in more thorough testing.
5.The nonincremental approach appears to use less machine time
6.At the beginning of the module-testing phase, there is moreopportunity for parallel activities if nonincremental testing is used
Given current trends in the computing industry,incremental testing is superior to non-incremental tesing.
Top-down Testing
The top-down strategy starts with the top, or initial, module in the program.
After testing the top module, numerous sequences are possible.Ingeneral, there is no best sequence, but here are two guidelines toconsider
1. If there are criticalsections of the program design the sequence such that these sectionsare added as early as possible.A “critical section” might be a complexmodule, a module with a new algorithm, or a module suspected to beerror prone.
2. Design the sequence such that the I/O modules are added as early as possible.
Bottom-up Testing
The bottom-up strategy begins with the terminal modules in the program(the modules that do not call other modules). After these modules havebeen tested, again there is no best procedure for selecting the nextmodule to be incrementally tested.
In most cases, driver modules are easier to produce than stub modules.
Chp 6 Higher-Order Testing
Function Testing
function testing is a process of attempting to find discrepancies between the program and the external specification
Except when used on small programs, function testing is normally ablack-box activity. That is, you rely on the earlier module-testingprocess to achieve the desired white-box logic-coverage criteria.
System Testing
System testing is the most misunderstood and most difficult testingprocess. System testing is not a process of testing the functions ofthe complete system or program, because this would be redundant withthe process of function testing.
system testing has a particular purpose: to compare the system or program to its original objectives.
Because of the absence of a methodology, system testing requires a substantial amount of creativity
1.Facility Testing
2.Volume Testing
System testing subjectsthe program to heavy volumes of data.The purposeof volume testing is to show that the program cannot handle the volumeof data specified in its objectives.
3.Stress Testing
Stress testing subjects the program to heavy loads or stresses.Thisshould not be confused with volume testing; a heavy stress is a peakvolume of data, or activity, encountered over a short span of time.
One of the common recipients of stress testing is Web-based applications.
4.Usability Testing
5.Security Testing
Security testing is the process of attempting to devise test cases that subvert the program’s security checks
6.Performance Testing
7.Storage Testing
8.Configuration Testing
9.Compatibility/Configuration/Conversion Testing
10.Installability Testing
11.Reliability Testing
12.Recovery Testing
13.Serviceability Testing
14. Documentation Testing
15.Procedure Testing
Acceptance Testing
Acceptance testing is the process of comparing the program to itsinitial requirements and the current needs of its end users.
It is an unusual type of test in that it usually is performed by theprogram’s customer or end user and normally is not considered theresponsibility of the development organization.
Installation Testing
It is an unusual type of testing because its purpose is not to findsoftware errors but to find errors that occur during the installationprocess.
Installation tests should bedeveloped by the organization that produced the system, delivered aspart of the system, and run after the system is installed.
Test Planning and Control
Regression testing is important because changes and error correctionstend to be much more error prone than the original program code
Test Completion Criteria
Thecompletion criteria typically used in practice are both meaningless andcounterproductive.The two most common criteria are these:
1. Stop when the scheduled time for testing expires.
2. Stop when all the test cases execute without detecting errors;that is, stop when the test cases are unsuccessful
Chp 7 Debugging
Of the two aspects of debugging,locating the error and correcting it,locating the error represents perhaps 95 percent of the problem.
Debugging by Brute Force
Brute force methods can be partitioned into at least three categories:
1. Debugging with a storage dump.
2. Debugging according to the common suggestion to “scatter print statements throughout your program.”
3. Debugging with automated debugging tools.
The general problem with these brute force methods is that they ignore the process of thinking.
Debugging by Testing
There are two types of test cases: test cases for testing, where the purpose of the test cases is to expose a previously undetected error, and test cases for debugging, where the purpose is to provide information useful in
locating a suspected error.
test cases for testing tend to be “fat” because you are trying to covermany conditions in a small number of test cases. Test cases fordebugging, on the other hand, are “slim” since you want to cover only asingle condition or a few conditions in each test case.
Error-Locating Principles
1.think.The most effective method of debugging is a mental analysis of the information associated with the error’s symptoms.
2.If You Reach an Impasse, Sleep on It.The human subconscious is a potent problem solver.
3.If You Reach an Impasse, Describe the Problem to Someone Else
4.Use Debugging Tools Only as a Second Resort
5.Avoid Experimentation—Use It Only as a Last Resort
Error-Repairing Techniques
1.Where There Is One Bug, There Is Likely to Be Another.In other words, errors tend to cluster
2.Fix the Error, Not Just a Symptom of It
3.The Probability of the Fix Being Correct Is Not 100 Percent
4.The Probability of the Fix Being Correct Drops as the Size of the Program Increases
5.Beware of the Possibility That an Error Correction Creates a New Error
6.Change the Source Code, Not the Object Code
Chp 8 Extreme Testing
The practice of creating unit tests first is the shining point of theXP methodology, as it forces you to understand the specification toresolve ambiguities before you begin coding.
Chp 9 Testing Internet Applications
three-tier client-server (C/S) architecture:Presentation，Business Logic，Data
Testing Internet-based applications is best tackled with a divide-and-conquer approach.
In a nutshell, presentation layer testing is very labor intensive.
Testing the end-user environment, also known asbrowser-compatibility testing, is often the most challenging aspect oftesting Internet-based applications
The Psychology of Testing
Testing is the process of executing a program with the intent of finding errors.
The most important considerations in software testing are issues of psychology,
Human beings tend to be highly goal-oriented, and establishing the proper goal has an important psychological effect.
Testing is a destructive process, even a sadistic process, which explains why most people find it difficult.
Atest case that finds a new error can hardly be consideredunsuccessful;rather, it has proven to be a valuable investment. Anunsuccessful test case is one that causes a program to produce thecorrect result without finding any errors.
Psychological studies tell us that people perform poorly when they setout on a task that they know to be infeasible or impossible.
Black-Box Testing(data-driven testing)
exhaustive input testing is impossible
White-Box Testing(logic-driven tesing)
exhaustive path testing, like exhaustive input testing, appears to be impractical, if not impossible.
Software Testing Principles
Principle 1: A necessary part of a test case is a definition of the expected output or result.
Principle 2: A programmer should avoid attempting to test his or her own program.
Principle 3: A programming organization should not test its own programs.
Principle 4:Thoroughly inspect the results of each test.
Principle 5:Test cases must be written for input conditions that are invalid and unexpected, as well as for those
that are valid and expected.
Principle 6: Examining a program to see if it does not do what it issupposed to do is only half the battle; the other half is seeingwhether the program does what it is not supposed to do.
Principle 7: Avoid throwaway test cases unless the program is truly athrowaway program.Saving test cases and running them again afterchanges to other components of the program is known as regression testing.
Principle 9:The probability of the existence of more errors in a section of a program is proportional to the number
of errors already found in that section.
Principle 10:Testing is an extremely creative and intellectuallychallenging task.It is probably true that the creativity required intesting a large program exceeds the creativity required in designing that program.
Chp 3 Program Inspections, Walkthroughs, and Reviews
nspections/walkthroughs and computer-based testing are complementary
The simpleact of reading aloud a program to an audience seems to be a remarkably effective error-detection technique.
For the inspection process to be effective, the appropriate attitude must be established.
Chp 4 Test-Case Design
Test-case design is so important because complete testing is impossible.
Chapter 2 showed that exhaustive black-box and white-box testing are,in general, impossible, but suggested that a reasonable testingstrategy might be elements of both.
The recommended procedure is to develop test cases using the black-box methods and then develop supplementary test cases as necessary with white-box methods.
Logic-Coverage Testing
If you back completely away from path testing, it may seem that aworthy goal would be to execute every statement in the program at leastonce. Unfortunately, this is a weak criterion for a reasonable white-box test.
A stronger logic-coverage criterion is known as decision coverage or branch coverage.
Decision coverage usually can satisfy statement coverage.Decisioncoverage is a stronger criterion than statement coverage, but it stillis rather weak. A criterion that is sometimes stronger than decisioncoverage is condition coverage.
condition coverage usually is superior to decision coverage.Althoughthe condition-coverage criterion appears, at first glance, to satisfythe decision-coverage criterion, it does not always do so.
The obvious way out of this dilemma is a criterion called decision/condition coverage. It requires sufficient test cases that each condition in a decision takes on all possible outcomes at least once, each decision takes on all possible outcomes at least once, and each point of entry is invoked at least once.
Equivalence Partitioning
you need to select the right subset, the subset with the highestprobability of finding the most errors. One way of locating this subsetis to realize that a well-selected test case also should have two other properties:
1.It reduces, by more than a count of one, the number of other testcases that must be developed to achieve some predefined goal of“reasonable” testing.
2.It covers a large set ofother possible test cases. That is, it tells us something about thepresence or absence of errors over and above this specific set of inputvalues
These two properties, although they appear to be similar, describe two distinct considerations.These two considerations form a black-box methodologyknown as equivalence partitioning.The second consideration is used todevelop a set of “interesting” conditions to be tested. The firstconsideration is then used to develop a minimal set of test casescovering these conditions.
Test-case design by equivalence partitioning proceeds in two steps:
(1) identifying the equivalence classes.
Given an input or external condition, identifying the equivalence classes is largely a heuristic process.
(2) defining the test cases.
The second step is the use of equivalence classes to identify the test cases. The process is as follows:
1. Assign a unique number to each equivalence class.
2. Until all valid equivalence classes have been covered by (incorporated into) test cases, write a new test case covering as many of the uncovered valid equivalence classes as possible.
3. Until your test cases have covered all invalid equivalence classes, write a test case that covers one, and only one, of the uncovered invalid equivalence classes.
Although equivalence partitioning is vastly superior to a randomselection of test cases, it still has deficiencies. It overlookscertain types of high-yield test cases, for example.
Boundary-Value Analysis
Experience shows that test cases that explore boundary conditions have a higher payoff than test cases that do not.
Boundary-value analysis differs from equivalence partitioning in two respects:
1.Rather than selecting any element in an equivalence class asbeingrepresentative, boundary-value analysis requires that one or moreelements be selected such that each edge of the equivalence class isthe subject of a test.
2. Rather than just focusingattention on the input conditions (input space), test cases are alsoderived by considering the result space (output equivalence classes).
Boundary-value analysis, if practiced correctly, isone of the mostuseful test-case-design methods. However, it often is usedineffectively because the technique, on the surface, sounds simple.
Cause-Effect Graphing
One weakness of boundary-value analysis and equivalence partitioning is that they do not explore combinations of input circumstances.
The testing of input combinations is not a simple task because even ifyou equivalence-partition the input conditions, the number ofcombinations usually is astronomical.
Cause-effect graphing aids in selecting, in a systematic way, ahighyield set of test cases. It has a beneficial side effect inpointing out incompleteness and ambiguities in the specification.
Cause-effect graphing requires the translation of a specification intoa Boolean logic network, it gives you a different perspective on, andadditional insight into, the specification. In fact, the development ofa cause-effect graph is a good way to uncover ambiguities andincompleteness in specifications.
Themost difficult aspect of the technique is the conversion of the graphinto the decision table. This process is algorithmic, implying that youcould automate it by writing a program
The Strategy
Thetest-case-design methodologies discussed in this chapter can becombined into an overall strategy. The reason for combining them shouldbe obvious by now: Each contributes a particular set of useful testcases, but none of them by itself contributes a thorough set of testcases.
Chp 5 Module (Unit) Testing
Test-Case Design
You need two types of information when designing test cases for a module test: a specification for the module and the module’s source code
Module testing is largely white-box oriented.
the multicondition-criterionis superior to the other criteria, and any logic-coverage criterion isnot good enough to serve as the only means of deriving module tests.
Incremental Testing
Thequestion pondered here is the following: Should you test a program bytesting each module independently and then combining the modules toform the program, or should you combine the next module to be testedwith the set of previously tested modules before it is tested
The testing of each module requires a special driver module and one or more stub modules.
1.Nonincremental testing requires more work.
2.Programming errors related to mismatching interfaces or incorrectassumptions among modules will be detected earlier if incrementaltesting is used.
3.debugging should be easier if incremental testing is used.
4.Incremental testing might result in more thorough testing.
5.The nonincremental approach appears to use less machine time
6.At the beginning of the module-testing phase, there is moreopportunity for parallel activities if nonincremental testing is used
Given current trends in the computing industry,incremental testing is superior to non-incremental tesing.
Top-down Testing
The top-down strategy starts with the top, or initial, module in the program.
After testing the top module, numerous sequences are possible.Ingeneral, there is no best sequence, but here are two guidelines toconsider
1. If there are criticalsections of the program design the sequence such that these sectionsare added as early as possible.A “critical section” might be a complexmodule, a module with a new algorithm, or a module suspected to beerror prone.
2. Design the sequence such that the I/O modules are added as early as possible.
Bottom-up Testing
The bottom-up strategy begins with the terminal modules in the program(the modules that do not call other modules). After these modules havebeen tested, again there is no best procedure for selecting the nextmodule to be incrementally tested.
In most cases, driver modules are easier to produce than stub modules.
Chp 6 Higher-Order Testing
Function Testing
function testing is a process of attempting to find discrepancies between the program and the external specification
Except when used on small programs, function testing is normally ablack-box activity. That is, you rely on the earlier module-testingprocess to achieve the desired white-box logic-coverage criteria.
System Testing
System testing is the most misunderstood and most difficult testingprocess. System testing is not a process of testing the functions ofthe complete system or program, because this would be redundant withthe process of function testing.
system testing has a particular purpose: to compare the system or program to its original objectives.
Because of the absence of a methodology, system testing requires a substantial amount of creativity
1.Facility Testing
2.Volume Testing
System testing subjectsthe program to heavy volumes of data.The purposeof volume testing is to show that the program cannot handle the volumeof data specified in its objectives.
3.Stress Testing
Stress testing subjects the program to heavy loads or stresses.Thisshould not be confused with volume testing; a heavy stress is a peakvolume of data, or activity, encountered over a short span of time.
One of the common recipients of stress testing is Web-based applications.
4.Usability Testing
5.Security Testing
Security testing is the process of attempting to devise test cases that subvert the program’s security checks
6.Performance Testing
7.Storage Testing
8.Configuration Testing
9.Compatibility/Configuration/Conversion Testing
10.Installability Testing
11.Reliability Testing
12.Recovery Testing
13.Serviceability Testing
14. Documentation Testing
15.Procedure Testing
Acceptance Testing
Acceptance testing is the process of comparing the program to itsinitial requirements and the current needs of its end users.
It is an unusual type of test in that it usually is performed by theprogram’s customer or end user and normally is not considered theresponsibility of the development organization.
Installation Testing
It is an unusual type of testing because its purpose is not to findsoftware errors but to find errors that occur during the installationprocess.
Installation tests should bedeveloped by the organization that produced the system, delivered aspart of the system, and run after the system is installed.
Test Planning and Control
Regression testing is important because changes and error correctionstend to be much more error prone than the original program code
Test Completion Criteria
Thecompletion criteria typically used in practice are both meaningless andcounterproductive.The two most common criteria are these:
1. Stop when the scheduled time for testing expires.
2. Stop when all the test cases execute without detecting errors;that is, stop when the test cases are unsuccessful
Chp 7 Debugging
Of the two aspects of debugging,locating the error and correcting it,locating the error represents perhaps 95 percent of the problem.
Debugging by Brute Force
Brute force methods can be partitioned into at least three categories:
1. Debugging with a storage dump.
2. Debugging according to the common suggestion to “scatter print statements throughout your program.”
3. Debugging with automated debugging tools.
The general problem with these brute force methods is that they ignore the process of thinking.
Debugging by Testing
There are two types of test cases: test cases for testing, where the purpose of the test cases is to expose a previously undetected error, and test cases for debugging, where the purpose is to provide information useful in
locating a suspected error.
test cases for testing tend to be “fat” because you are trying to covermany conditions in a small number of test cases. Test cases fordebugging, on the other hand, are “slim” since you want to cover only asingle condition or a few conditions in each test case.
Error-Locating Principles
1.think.The most effective method of debugging is a mental analysis of the information associated with the error’s symptoms.
2.If You Reach an Impasse, Sleep on It.The human subconscious is a potent problem solver.
3.If You Reach an Impasse, Describe the Problem to Someone Else
4.Use Debugging Tools Only as a Second Resort
5.Avoid Experimentation—Use It Only as a Last Resort
Error-Repairing Techniques
1.Where There Is One Bug, There Is Likely to Be Another.In other words, errors tend to cluster
2.Fix the Error, Not Just a Symptom of It
3.The Probability of the Fix Being Correct Is Not 100 Percent
4.The Probability of the Fix Being Correct Drops as the Size of the Program Increases
5.Beware of the Possibility That an Error Correction Creates a New Error
6.Change the Source Code, Not the Object Code
Chp 8 Extreme Testing
The practice of creating unit tests first is the shining point of theXP methodology, as it forces you to understand the specification toresolve ambiguities before you begin coding.
Chp 9 Testing Internet Applications
three-tier client-server (C/S) architecture:Presentation，Business Logic，Data
Testing Internet-based applications is best tackled with a divide-and-conquer approach.
In a nutshell, presentation layer testing is very labor intensive.
Testing the end-user environment, also known asbrowser-compatibility testing, is often the most challenging aspect oftesting Internet-based applications
522
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
