软件测试中的错误缺陷失效故障术语解析

软件测试中对软件错误的称谓很多，例如错误，缺陷，失效，故障，事故，异常等，对应着Mistake，defect, failure, fault, incident, exception等。这些概念的定义是什么，有什么区别与联系，本文将以IEEE和ISO的术语定义为基础，阐述软件生命周期中各个术语的演变关系。

This article will be useful for those who are looking for a clear difference between the different terminologies that are used for the term 'defect' in the SDLC.

The main intention of this article is to bring a clear picture about the 'mistake' committed by an analyst in the early stages of the  SDLC, which leads to a 'defect' in the final stage of the SDLC. I would also suggest using the relevant term in the respective phase of the SDLC, which would be beneficial for process improvement and for calculating metrics.

In this context, the concept of 'Mistake Metamorphism' is explained by taking the definitions given by IEEE and ISO and by streamlining them by mapping them to the respective phases of the SDLC.

 

Introduction

In software development projects, a 'mistake' or 'fault' can be committed at any stage during development. For each phase, we have different techniques for detecting and eliminating the mistake faults that originate in that phase. However, no technique is perfect, and it is expected that some of the mistakes/faults of the earlier phases will finally manifest themselves in the code. This is particularly true because in the earlier phases most of the verification techniques are manual as long as no executable code exists.Ultimately, these remaining mistakes/faults will be reflected

in the code and continue into the subsequent phases of the SDLC.

Why does software have defects? The reasons include:

• Miscommunication
• Inappropriate skills
• Complex technology
• Programming errors
• Tight schedules
• Poorly documented code
• Changing requirements
• Failure to communicate andact as a team
• Lack of domain skills

When do software defects occur?

• The software doesn't do something that the specification says it should do
• The software does something that the specification says it shouldn't do
• The software does something that the specification doesn't mention

The software doesn't do something that the specification doesn't mention but should; the software is difficult to understand, hard to use, slow, or– in the software tester's eyes – will be viewed by the end user as just plainly not right.

Mistake Metamorphism:

The word“metamorphism” comes from Greek, whereby meta = change, morph = form, so metamorphism means to change form. We can use this word in software engineering for the evolution of a defect in the final stage of software deployment. The main intention is to show a clear picture about the 'mistake' committed by an analyst in the early stages of the SDLC, which leads to a 'defect' in the final stage of the SDLC.

Different stages of a 'mistake' in the entire SDLC can be describedby using the following terms:

• Mistake
• Anomaly
• Fault
• Failure
• Error
• Exception
• Crash
• Bug
• Defect
• Incident
• Side effect

You might be amazed that so many names could be used to describea software failure. Why so many? If you look up these words in the English Dictionary, you'll find that they all have slightly different meanings.

But in the virtual software dictionary, the name of a software failure will be changed based on the phase in the SDLC; this is what we are going to prove.

figure 01

The high level view (figure 01) tells how a 'mistake' committed by an analyst leads to a 'defect' in the software product/project, once it has been delivered to the client. We will call this a 'Mistake Metamorphism'.

Let's see the example below which describes in more detail how this 'mistake' committed by an Analyst in the initial stage of the SDLC leads to a 'Defect' in the software project/product.

The diagram (figure 02) depict the transformation of a 'Mistake'to a 'Defect'

An Analyst commits a 'mistake' which leads to an anomaly:

a)      The customer explained the requirement(s) and gave business requirements documents (BRDs) to the analysis team.

b)     In the requirements phase of the project,the analyst(or analysts) will prepare a functional specification from the customer's BRDs. At this time they may make mistakes n any of the following ways:

figure 02

i)       Misunderstanding the business requirement

ii)     Mistakes in the documentation

iii)   Inconsistency in the functional specification

Let's correlate the above points (a) and (b) with the definition that is provided by IEEE for the term 'mistake'.

Mistake (definition by IEEE):

A human action that produces an incorrect result, and the subject of the quality improvement process.

The information provided as examples in (a) and (b) precisely reflect the IEEE definition of 'mistake'.

c)      If the 'mistake' committed in the functional specification document is identified by the same analyst, another analyst, a designer, developer or a test engineer, then this mistake can be called an 'anomaly'.

Let's correlate the above point (c) with the definition that is provided by the IEEE for the term 'anomaly'.

Anomaly (definition by IEEE):

Anything observed in the documentation or operation of software that deviates from expectations based on previously verified software products or reference documents.

The information provided in point (c) precisely reflects the IEEE definition of 'anomaly'.

A designer commits a 'mistake' which leads to an anomaly:

d)     As part of the design activities in the design phase, software designers will prepare system design and detailed design documents. If the 'anomaly' is not caught by any of the people in the project during the design phase, the same 'mistake' will be carried forward.

A developer commits a 'fault' or 'mistake' which leads to failure:

e)      In the coding phase, developers write the code as per the design documents. Developers may make any of the following mistakes:

i)       There may be typing errors

ii)     Incorrect logic

iii)   Cross-Site Scripting

iv)   Choosing the wrong database, etc.

A mistake committed by the developer in the code can be called a 'fault'.

Let's correlate point (e) with the definition that is provided by the IEEE for the term 'fault'.

Fault (definition by IEEE):

An incorrect step, process, or data definition in a computer program which causes the program to perform in an unintended or unanticipated manner.

The information provided in point (e) precisely reflects the IEEE definition of 'fault'.

Once the coding has been completed, the code will be reviewed/ tested as part of unit testing. If the 'fault' is identified in unit testing, then that 'fault' is called a 'failure'.

Let's correlate the same with the definition that is provided by the IEEE for the term 'failure'.

Failure (definition by IEEE):

The inability of a system or component to perform its required functions within specified performance requirements.

A 'fault' identified as part of a review or unit testing exactly resembles the definition that is provided by the IEEE for 'failure'.

Any faults that are not identified in unit testing will be carried forward and can then be identified in integration or system testing. At this stage, the identified fault(s) is/are called

6.      Error

7.      Exception

8.      Crash

9.      Bug

Let's first discuss the above points one by one.

1. Error:Errors type are:

• Run-time error,
• Syntax error, and/or
• Semantic error.

These errors will be thrown by the system if there is discrepancy in the code.

Run-time error, semantic error– an error in logic or arithmetic that must be detected at run time.

Syntax error– an error of language resulting from code that does not conform to the syntax of the programming language;“syntax errors can be recognized at compilation time”;“a common syntax error is to omit a parenthesis”.

Let's look at the definitions provided by ISO.

Error

1)     (ISO) a discrepancy between a computed, observed, or measured value or condition and the true, specified, or theoretically correct value or condition;

2)     A quality problem discovered by software engineers (or others) before the software is released to the end-user (or to another activity in the software process). [Pressman, 203]

The above definitions exactly resemble the types of error that are identified in system/integration testing.

1. Exception: There are some faults that occur in rare conditions because of problems in hardware, impact of power, unintentional memory consumption (i.e., memory leaks). These types of faults are called 'exceptions'.

Note:Memory Leak: A bug in a program that prevents it from freeing up memory that it no longer needs. As a result, the program grabs more and more memory until it finally crashes because there is no more memory left.

Here is the definition that is provided by the IEEE for 'exception', let's correlate the same with the above cited point.

Exception (definition by IEEE):

An event that causes suspension of normal program execution; types include addressing exception, data exception, operation exception, overflow exception, protection exception, and underflow exception.

The above definition exactly resembles the 'exception' that is identified in system/integration testing.

2. Crash:All the above types of faults (i.e., failure, error and exception) do not impact the software that much. There will be almost no any data loss or software crash. A fault that is caused because of fire, accident, natural disaster, etc. will, however, lead to a complete failure of the software. In most cases, the system cannot be recovered. This type of failure is called a 'crash'.

Here is the definition of 'crash' given by the IEEE:

Crash (definition by IEEE):

The sudden and complete failure of a computer system or component.

Let's elaborate point 4, i.e., 'bug' in more detail.

A tester might miss the 'fault' or 'mistake':

j)       If there is a fault in the code that may not be thrown as an error to the user, then this type of fault is called a bug. Bugs can be

i)       Programming errors,

ii)     Computer error, and/or

iii)   Functional cohesion.

i)       Programming error: An error resulting from bad code in some program involved in producing the erroneous result.

ii)     Computer error, error (computer science): The occurrence of an incorrect result produced by a computer

iii)   Functional cohesion: A type of cohesion in which the tasks performed by a software module all contribute to the performance of a single function.

These bugs will be identified by the test team during integration or system testing.

Let's look at the definition for the term 'bug' provided by the IEEE.

Bug (definition by IEEE):

（1）      Things the software does that it is not supposed to do, [or] something the software doesn't do that it is supposed to. [Telles and Hsieh, 5];

（2）      A fault in a program which causes the program to perform in an unintended or unanticipated manner.

Note:This term is one of the oldest in the software profession. It originated in the 1950's when the late Admiral Grace Hopper found an actual insect hamming the contacts of an electromechanical switch. The term now refers to errors or defects that find their way into programs and systems. Some software quality researchers find it desirable to replace the pejorative term“bugs” with other terms such as errors, defects, faults, and failures.

The above definition exactly resembles the types of faults that are identified in system/integration testing.

A missed 'error', 'exception', 'crash', or 'bug' in testing will lead to a 'defect' in production:

k)     If the faults are not identified prior to the software release, then they are identified once the software has been released to customer after its deployment in the production environment. If the faults are identified during beta-testing, or user acceptance testing, or in production (here support will be considered for a certain period based on the contract), then these faults are called 'defects'.

Here are the different definitions that are provided for the term 'defect':

Defect

1）             (IEEE) a product anomaly;

2）             Synonymous with fault, both imply a quality problem discovered after the software has been released to end-users (or to another activity in the software process); [Pressman, 203]

3）             fault; [SWEBOK, Ch.5, p.6]

4）             error, fault, or failure; [SWEBOK, Ch.11, p.9]

5）             non-conformance to requirements.

As we are trying to prove that #the terms for software failure will change based on the phase in the SDLC, in the virtual software dictionary, let's consider the definition provided by Pressman for 'defect'.

The above definition exactly resembles the fault(s) that is/are identified in post deployment.

Though the maintenance/sustenance phase is segregated, we are aware of the fact that once the software has been deployed, it will be in the maintenance phase. During this phase there won't normally be any deployments or upgrades to the software, except for hot fixes for the defects that are identified in the maintenance/ sustenance phase. Here complete support will be provided by the team that developed the software or by the vendor who took it over from the development team/vendor. Any issue that is identified in the maintenance/sustenance phase can be called an 'incident', which may or may not be a problem/issue. This is called a 'ticket' by most people. The issue could be due to a lack of user training on the software, to environmental issues or due to non- reproducible issues or some other software compatibility issues, etc.

Here is the definition that is provided by the IEEE for the term 'incident'.

Incident (definition by IEEE):

An incident is an event occurring during testing that requires attention (investigation). [IEEE 1008] Incidents do not necessarily justify the filing of a formal problem report. The definition of the IEEE exactly resembles the occurrence of issues in the maintenance phase.

Issues that are identified and fixed in production (during the maintenance phase) lead to 'side effects':

l)       l) Updating/fixing any of the issues identified in the maintenance phase (i.e., anomaly, failure, error, exception, crash, bug, defect) may impact other areas of the updated functionality/code, which leads to a new issue called a 'side effect'.

Side effect (definition by IEEE):

An unintended alteration of a program's behavior caused by a change in one part of the program, without taking into account the effect the change has on another part of the program.

(…)

References:

Standards:

IEEE Standard 610.12-1990. IEEE Standard Glossary for Software Engineering Terminology.

IEEE Trial Draft SWEBOK Version 1.00. Software Engineering Bodyof Knowledge.

Books:

Pressman, Roger S. Software Engineering: A Practitioner's Approach.5th ed. New York: McGraw-Hill, 2001.

Pankej Jalote An Integrated Approach to Software Engineering, 2^nd Edition.

Biography

Trinadh Kumar Bonam

Mr. Trinadh Kumar Bonam, MSc (Computer Science), Mtech (Software Engineering), Six Sigma (GB), Certified Scrum Master, HNC (from NIIT), ISTQB® foundation, has worked as test engineer and as test lead. He has seven years of testing experience in domains like sales/partner sales, call center, telecommunications (OSS/BSS/VAS), retail, manufacturing, etc.

As a Test Analyst, he has worked in various projects covering multiple technologies (like Siebel, MS CRM, etc.) and has been involved in agile testing.

He now works for the established No. 1 company in India. His primary objective is‚QUALITY'. He is interested in learning new things, he finds never ending challenges in testing. He can be reached at trinadh.bonam@gmail.com, and you can find more about him on LinkedIn.

 

编辑简介：崔启亮，北京昱达环球科技有限公司ISTQB培训讲师，CSTQB专家组成员，电子邮件gcui@igsgroup.com.cn