Software Testing: Theory on Defect Detection

Theory: A defect can only be discovered in an environment that contains the defect. 

This seems very obvious! So why even bother to mention it, let alone post a blog entry about it?  The motivation for this entry comes from a familiar situation that i am sure most testers would have encountered. 

Software testers devise extensive sets of tests and execute them prior to a product's release. These tests are usually executed on setups that testers have prepared in their lab environment. Many organizations may have invested significant sums of money to have the lab infrastructure in place. However, what generally tends to happen is that when this product is released, the customer reports issues pretty quickly. So, what happened? What happened to all the man-hours spent on testing and the investment on expensive lab equipment? Why did our in-house testing efforts not show up these defects that a customer seemed to find "easily"? What are we doing wrong?

This brings us to our theory, i.e. a defect can only be discovered in a system or environment that contains the defect . A defect that might show up in a customer environment may not manifest itself in a sanitized lab environment. Often our lab environment is setup and controlled based on our view of how the product is likely to be used. Within the confines of the boundaries we have defined, we execute our battery of tests and feel confident when the tests run without reporting issues. However, a customer's environment suffers from no such boundary constraints and does not find it hard to expose a defect. Defects may not necessarily be in the product itself. It could arise from the interactions of the product with its operating environment, dependencies, usage, etc.

Therefore, unless we are able to replicate in sufficient detail the customer environment and the likely real-world usage scenarios after the product is released, we will likely continue to see an increasing trend of customer reported issues.