1)Validation vs. Verification
OrVerification vs. Validation.
Two terms that sometimes confuse software test engineers are "validation" and "verification". Let me get to the bottom line: in a software testing context, although technically the two terms may have slightly different meaning, there is no practical difference between the two. Both validation and verification loosely mean, "Does the program/system under test do what it's supposed to do?" And that means, "Does the SUT meet its design, requirements, and other specifications?" If you look up validation and verification in an ordinary English dictionary, you'll find that they mean almost exactly the same thing. In fact, the definition of validation often contains the term "verification" and the definition of "verification" often contains the term "validation". For example, the Merriam-Webster dictionary gives the synonym for the root word "verify" as "confirm", and gives the synonym for the root word "validate" as "confirm" too. And the Oxford English Dictionary defines "validate" as, "demonstrate that something is true", and defines the term "verify" as, "demonstrate that something is true". In other words, in ordinary English usage the terms "validate" and "verify" have exactly the same meaning.
Somewhere along the line, in the field of software testing (or software quality assurance if you want to get more fancy), someone decided that the two terms should have slightly different meanings. The IEEE, an organization which has defined many terms gives this:
Validation - Confirmation by examination and provision of objective evidence that the particular requirements for a specific intended use are fulfilled.
Verification - Confirmation by examination and provision of objective evidence that specified requirements have been fulfilled.
Obviously the two IEEE definitions are so close to each other it's hard to determine the difference. These definitions are often interdivted as suggesting that validation means, "Does the system under test satisfy user specifications?", and that verification means, "Does the system under test satisfy a functional specification?" Sometimes this is put informally as, "validation means did we build the right product?" and "validation means did we build the product right?" However, at least in practice, the two terms have been used interchangeably for so long there is little chance of changing this common usage.
Now, with all that said, I've got to confess that I use the two terms somewhat differently. In general I use the term "verification" to mean, "The process of determining if it works correctly?" And I often use the term "validation" to mean, "The process of double-checking my test results". For example, when testing a Web Service, you can write a lightweight test harness which sends input to and fetches the return value from the Web method under test using HTTP. You can then write a second lightweight harness which does the same thing using TCP/IP. You should of course get the same results, validating (in my sense of the word) your test results.
My point is this: in practical usage the terms "validation" and "verification" both mean, "Does the system under test work correctly?" If you intend to use the terms differently in conversation or a written document of some sort, you should explicitly define your local meanings of the terms.
If you enjoyed reading this blog entry, you might enjoy working at Microsoft. My parent company, Volt Information Sciences, is always looking for software engineers. Check out the job postings at http://jobs.volt.com/.
2)Monkey Tests
Monkey testing should be a part of every significant software testing effort. Monkey testing is the process of feeding random input to your system under test. For example, if you were applying monkey testing to a simple calculator program, you would randomly hit a random number of various input keys. The origin of the term monkey testing derives from the Infinite Monkey Theorem which states that a monkey hitting typewriter keys at random will almost surely eventually produce every book in the French national library. The theorem is derived from a similar statement in a 1913 book by French author Emil Borel. The phrase was Anglicized in the 1970s to variations like, "an infinite number of monkeys typing for an infinite amount of time will eventually produce all the works of William Shakespeare."
Although manual monkey testing is possible, monkey testing is almost always performed using automation. It turns out that it quite useful to classify monkey testing into either two or three categories based on how intelligent the virtual monkey is. The simplest type of monkey testing is "dumb monkey" testing. In dumb monkey testing, the virtual monkey test automation knows almost nothing about the system under test (SUT). The monkey test program simply feeds random input to the system under test. For example, suppose the SUT has a text box which is designed to accept a user-supplied URL as input, like "http://www.vteOnline.com". A dumb monkey test program supplies random characters to the text box, not even attempting to make a valid URL, and then submits the characters. Dumb monkey testing has the advantage that it is relatively cheap to implement. However, dumb monkey testing does not have the ability to compare an expected result with an actual result, so dumb monkey testing will only uncover bugs which produce a crash or hang in the SUT.
In "smart monkey" testing, the test automation has some knowledge of the SUT inputs, states, and expected results. For instance, in the divvious URL-input example, a smart monkey test may know the structure of a valid URL and expect different kinds of results for valid and invalid inputs. Smart monkey systems are much more difficult to create but can uncover bugs other than hanging and crashing bugs.
Of course, the labels "smart" and "dumb" monkey tests are arbitrary. Monkey tests can take on a continuum of sophistication. A "brilliant monkey" test system might use some rudimentary artificial intelligence to more closely mimic realistic user input. For instance, you could write a monkey system which knows that certain letters on a keyboard which are physically close together are often accidentally switched by human users.
Based on my experience, I've found that relatively dumb monkeys give you the best test-bang for the test-buck. In a tight development cycle, the time cost of developing a smart monkey is usually too high for the quality improvement payoff.
If you enjoyed reading this blog entry, you might enjoy working at Microsoft. My parent company, Volt Information Sciences, is always looking for software engineers. Check out the job listings at http://jobs.volt.com/.
3)Smoke Tests
Years ago, one of the new terms I heard when I started working at Microsoft as a software test engineer was "smoke test". A smoke test is a quick-and-dirty test designed to determine if the software program/system under test has any major functionality bugs. or put slightly differently, a smoke test is an ad hoc test, usually performed manually, intended to uncover only serious, high severity bugs. A smoke test is most often performed right after a developer adds new code to the system under test, and before the new code is checked into the main code base in the source control system. You can think of smoke tests as an informal type of DRT (Developer Regression Test).
Over time, I heard several theories about the origin of the term smoke test. After some research, I determined that there are at least four possible origins. In hardware testing, the term smoke test means the process of testing a new piece of hardware simply by plugging it into an electrical source and powering the device up. If the device smokes -- actually starts emitting smoke -- then this is not a good sign and the device fails the smoke test. If no smoke appears, the device passes the smoke test.
A second possible origin of the term smoke test comes from the field of plumbing. After creating a plumbing or ventilation system by hooking pipes together, a technician pumps smoke into the system and then looks for leaks.
A third possible origin of the term comes from printing. In the days before computer typography, printing was done by stringing together individual pieces of metal with raised imdivssions -- one piece of type for every letter. If you've every seen an old typewriter you'll know exactly what I mean. The pieces of type would be coated with ink and then divssed onto paper. If you think about it for a moment you'll realize that each piece of metal must have the letter it will print in reversed form. So, the process of creating metal type involved carving letters in reverse. To test how the letter would look when used for printing, the type-maker could light a candle, let smoke coat the metal type piece and then divss the piece onto a piece of paper to see the resulting imdivssion.
The fourth possible origin of the term smoke test comes from the old saying, "where there's smoke, there's fire." And that term, or close to it, exists in many different languages and has been around as far back as I could research.
Like most word origins, the truth is probably that the term smoke test comes from more than one source. You can see that all four possible origins involve physical smoke, and are tests of some sort. I suspect that multiple people started using the term more or less simultaneously when the first high-level programming languages (Fortran, COBOL, BASIC) appeared in the late 1950s and early 1960s. One thing is for sure though: if you add new code to your program, and then run the program and your computer starts smoking, you've written some divtty cool code!
4)BVTs and DRTs
The terms BVT and DRT are important ones to software test engineers. BVT stands for Build Verification Tests, and DRT stands for Developer Regression Tests. BVTs are a set of tests which are run immediately after a new build of the system under development has completed. The BVT test cases are designed to determine if the new build can be installed and tested by the test team. DRTs are a set of tests which are run immediately before new code is checked into the system under development. The DRT test cases are designed to determine if the new code breaks existing functionality or not.
Let me put it slightly differently, in chronological order. First a developer writes new code. Before checking the new code into a central source/version control system, the developer creates a private build of the system with the new code. The developer then runs a DRT test suite. If the private build passes the DRT suite, then the developer checks the new code into the central build system. Towards the end of the day (or week or some other timeframe depending on the frequency of builds), the Build Team builds a new version of the system under development which contains all the new code checked in by developers that day. After the build process completes (typically early in the morning of the next day), the Build Team runs the BVT test suite. If the new build passes the BVT tests, then the new build is "released to test", meaning a message is sent to the Test Team telling the testers where the new build is, and that the build is installable and testable.
In general, both BVTs and DRTs consist of automated test cases. Manual testing is just too time-consuming in a tight development cycle. There is often overlap between BVT and DRT test cases, meaning that some or all of the test cases in each test suite are often the same. In other words, the Test Team creates a large number of test cases. BVTs and DRTs are subsets of the entire set of test cases. Other common subset suites include DTR (daily test run), WTR (weekly test run), MTR (milestone test run), and FTP (full test pass).
5)Code Coverage
Code coverage is a software testing activity which is intended to measure how effective your test effort is. In other words, code coverage does not directly test the program/system you are testing. Code coverage analysis is the process of monitoring which parts of your system under test (SUT) are exercised by your test suite. (A test suite is a collection of individual test cases). For example, suppose you have a test suite which consists of 10,000 test cases. And suppose your SUT has 250,000 statements. In a code coverage situation you would run your test suite. Now suppose that the tests touch 200,000 statements, meaning that 200,000 statements are executed during the test suite run. In this situation your code coverage is 200,000 / 250,000 = 80. You now know that certain parts of your SUT aren't being tested at all! So, you can create new test cases. It is not uncommon for a test team to be fairly confident that they have good test suites, but after a code coverage analysis they discover they only have around 60-70 coverage. This has happened to me before.
It should be clear that code coverage is an absolutely essential part of any significant test effort.
Code coverage can operate at different levels of granularity. For instance, you can perform code coverage at the statement level, checking what percent of SUT statements are exercised by your test suite. or you can perform code coverage at the block level, checking what percent of the blocks of SUT code (typically code between { and } in languages such as C/C , C#, and Java). or you can perform code coverage at the function/method level, just checking to see what percentage of functions/methods in the SUT are entered.
Code coverage is simple in principle, but difficult in practice. There are several approaches. One approach is to instrument the source code of the SUT by inserting special code which logs entry and exit events as the code executes. This idea sounds good but doing this manually is very time-consuming, and instrumenting source code programmatically is extremely difficult -- you're essentially writing a complete language parser. Another approach is to create a service which monitors the run time activities of the machine/CPU running the tests. This is really, really difficult. In a .NET and Java environment you can attempt to instrument the Intermediate Language (.NET) or Byte Code (Java) which is a tiny bit easier, but still really difficult. There are other approaches too but they're all hard. I'm a big believer in lightweight, custom test automation, but code coverage is one area where using a commercial tool often makes sense.
You can read about code coverage in the April 2004 issue of MSDN Magazine at http://msdn.microsoft.com/msdnmag/issues/04/04/CodeCoverageAnalysis/.
| 正在装载数据…… |
6)Software Testing vs. Analysis
One software testing concept that beginning software test engineers often don't fully understand is the difference between testing and analysis. A software test case is an entity (for lack of a better word) which has a pass or a fail result -- nothing else. Software test analysis is an activity which results in information about the system under test. Here's an example of what I mean. Suppose you are testing the performance of some application program which searches through a database store of book title information. You exercise the application under a set of fixed conditions with a particular input which redivsents part of a title (say, "football") and discover that results (say, the data for 3 different books) are returned in 1.50 seconds. This is analysis, not testing. A test case on the other hand would specify a certain time value that if results are returned in that time or less, then the test case result is a pass, but if results are returned in longer than the specified time value, then the test result is fail.
Analysis and testing are most often confused when looking at SUT performance, as in the example above. However, the principle applies to all types of testing, including functionality testing. For example, suppose you are testing a calculator program. You enter 3 into the calculator and then hit the Square Root key, and 1.73205 is displayed in the answer area. This is analysis. A test case might look like:
0001 3 SqRt 1.73204999
where the four fields redivsent a test case ID, two test case inputs, and the expected result. If you got 1.73205, then this would be a fail result because the actual result (1.73205) does not exactly match the expected result (1.7320499).
Now obviously, this is a question of semantics. But details are important in software testing. And analysis and testing go hand in hand. For example, especially with performance testing, you must sometimes do some divliminary performance analysis to determine the threshold values for your test cases.
7)Agile Programming vs. Extreme Programming
Agile Programming and Extreme Programming are two types of software development methodologies. There are many slight variations on both Agile and Extreme programming. The key idea of Agile Programming is to construct software in a series of short (typically two to four weeks) iterations. Each iteration has its own planning, design, coding, testing, and release phases. Agile Programming emphasizes quick changes to design and verbal, rather than written, communication among team members. Agile Programming is quite similar to the Spiral software development methodology except that Agile has shorter iteration times. Agile Programming is particularly effective on smaller projects (say, 10 or fewer team members), non-critical projects (in other words, not projects such as medical or weapons systems), and non-distributed systems (in other words, when all team members are in one physical location).
Extreme Programming is most often considered a particular form of Agile Programming, although some engineers divfer to think of Extreme Programming as a separate methodology altogether. Extreme Programming is more specific the Agile Programming but in practice no two groups which use Extreme Programming seem to agree on exactly which characteristics are essential. From a practical point of view, Extreme Programming encourages (or requires depending on your point of view) "Pair Programming" where developers always work in pairs, and "Test-Driven Development" where unit tests are always created in conjunction with the system under development. Extreme Programming may be a good choice for prototype technologies and research projects.
You can contrast Agile Programming and Extreme Programming with the Waterfall model of software development. This methodology says, in essence, to start with anextensive requirements analysis, then thorough design, followed by coding, testing, and so on until finished. The Waterfall model is often used in huge military projects with long development times. Ultimately "Agile Programming", "Extreme Programming", "Spiral Development", and the "Waterfall Methodology" are all just labels. In reality most software projects use certain aspects of all these methodologies, all of which are based on sound software engineering principles and practices. During my time at Microsoft, I worked on thirteen different products and the most important part of success (and failure for certain projects) was not a particular methodology, but clear communication, hard work, and common sense.
8)Stress Testing vs. Load Testing
The terms "stress testing" and "load testing" are often used interchangeably by software test engineers but they are really quite different. Even the generally reliable wikipedia slightly botches the explanation of these two terms. Stress testing is the process of subjecting your program/system under test (SUT) to reduced resources and then examining the SUT's behavior by running standard functional tests. The idea of this is to expose problems that do not appear under normal conditions. For example, a multi-threaded program may work fine under normal conditions but under conditions of reduced CPU availability, timing issues will be different and the SUT will crash. The most common types of system resources reduced in stress testing are CPU, internal memory, and external disk space. When performing stress testing, it is common to call the tools which reduce these three resources EatCPU, EatMem, and EatDisk respectively.
Load testing is the process of subjecting your SUT to heavy loads, typically by simulating multiple users, where "users" can mean human users or virtual/programmatic users. The most common example of load testing involves subjecting a Web-based or network-based application to simultaneous hits by thousands of users. This is generally accomplished by a program which simulates the users. There are two main purposes of load testing: to determine performance characteristics of the SUT, and to determine if the SUT "breaks" gracefully or not. In the case of a Web site, you would use load testing to determine how many users your system can handle and still have adequate performance, and to determine what happens with an extreme load -- will the Web site generate a "too busy" message for users, or will the Web server crash in flames?
You can read about a memory stressor (EatMem) in the May 2006 issue of MSDN Magazine at http://msdn.microsoft.com/msdnmag/issues/06/05/TestRun/default.aspx. If you enjoyed this blog entry, you might enjoy working at Microsoft. My parent company, Volt Information Sciences, is always looking for software engineers. Check the job listings at http://jobs.volt.com/.
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
