Exploring Explainability

2021 IEEE 29 th International Requirements Engineering Conference (RE)

2021 IEEE 29th International Requirements Engineering Conference (RE) | 978-1-6654-2856-9/21/$31.00 ©2021 IEEE | DOI: 10.1109/RE51729.2021.00025

Exploring Explainability:

A Definition, a Model, and a Knowledge Catalogue

Larissa Chazette*, Wasja Bruni Ute" Timo Speithz

*Leibniz University Hannover, Software Engineering Group, Hannover, Germany

"Leibniz University Hannover, Cluster of Excellence PhoenixD, Hannover, Germany

"Saarland University, Institute of Philosophy and Department of Computer Science, Saarbrucken, Germany

Email: {larissa.chazette, wasja.brunotte}@inf.uni-hannover.de, timo.speith@uni-saarland.de

Abstract—The growing complexity of software systems and the influence of software-supported decisions in our society awoke the need for software that is transparent, accountable, and trustworthy. Explainability has been identified as a means to achieve these qualities. It is recognized as an emerging non-functional requirement (NFR) that has a significant impact on system quality. However, in order to incorporate this NFR into systems, we need to understand what explainability means from a software engineering perspective and how it impacts other quality aspects in a system. This allows for an early analysis of the benefits and possible design issues that arise from interrelationships between different quality aspects. Nevertheless, explainability is currently under-researched in the domain of requirements engineering and there is a lack of conceptual models and knowledge catalogues that support the requirements engineering process and system design. In this work, we bridge this gap by proposing a definition, a model, and a catalogue for explainability. They illustrate how explainability interacts with other quality aspects and how it may impact various quality dimensions of a system. To this end, we conducted an interdisciplinary Systematic Literature Review and validated our findings with experts in workshops.

Index Terms—Explainability; Explanations; Explainable Artificial Intelligence; Interpretability; Non-Functional Requirements; Quality Aspects; Requirements Synergy; Software Transparency

• I. Introduction

We live in the age of artificial intelligence (AI). Software decision-making has spread from simple daily decisions, such as the choice of a navigation route, to more critical ones, such as the diagnosis of cancer patients [2]. Systems have been strongly influencing various aspects of our lives with their outputs but can be as mysterious as black boxes to us [3]. This ubiquitous influence of black-box systems has induced discussions about the transparency and ethics of modern systems [4]. Responsible collection and use of data, privacy, and safety are just a few among many concerns. It is crucial to understand how to incorporate these concerns into systems and, thus, how to deal with them during requirements engineering (RE).

Explainability is increasingly seen as the preferred solution to mitigate a system’s lack of transparency and should be treated as a non-functional requirement (NFR) [5]. Incorporating explainability can mitigate software opacity, thereby helping users understand why a system produced a particular result and supporting them in making better decisions. Explainability also has an impact on the relationship of trust in and reliance on a system [6], and it may avoid feelings of frustration [7].

Although explainability has been identified as an essential NFR for software-supported decisions [8] and one of the pillars for trustworthy AI [9], there is still a lack of an extensive overview that investigates the impact of explainability on a system.

In this paper, we investigate the concept of explainability and its interaction with other quality aspects. We use the notion of quality aspects to refer both to NFRs and to aspects that relate to or compose NFRs. In this regard, we follow Glinz and see NFRs as attributes of or constraints on a system [10].

Previous studies have shown that explainability is not only a means of achieving transparency and building trust but that it is also linked to other important NFRs, such as usability and auditability [11]—[13]. Explainability can, however, have both a positive and a negative impact on a system. Like other NFRs, explainability is difficult to elicit, negotiate, and validate. Eliciting and modeling NFRs is often a challenge for requirements engineers due to the subjective, interactive, and relative nature of NFRs [14]. Often there are trade-offs between NFRs in a system that must be identified and resolved during the requirements analysis [14], [15].

One of the challenges with respect to NFRs stems from the fact that information concerning them is rather tacit, distributed, and based on experience [16], [17]. To mitigate this, a usual strategy adopted by requirements engineers to deal with NFRs during RE is to make use of artifacts such as conceptual models [16], pre-established lists, or knowledge catalogues [14]. Models and catalogues can be used to help specify quality requirements [18]. They may compile knowledge about specific NFRs and their interactions with other quality aspects. Among others, such artifacts support the elicitation process. Models can be used to understand the taxonomy of a given quality aspect during analysis, while catalogues can support the analysis of trade-offs where it is essential to understand how two or more NFRs will interact in a system and how they can coexist [19]. Requirements engineers can also use models and catalogues to ask stakeholders about their interest with respect to the general system quality [20]. Existing works propose to build such artifacts to capture and structure knowledge that is scattered among several sources [14], [16], [21], [22].

Since explainability is an emerging requirement, there is still a lack of structured knowledge about this NFR. To bridge this gap, we employed a multi-method research strategy consisting of an interdisciplinary Systematic Literature Review (SLR) and workshops. Overall, our goal is to advance the knowledge towards a common terminology and semantics, facilitating the discussion and analysis of explainability during the RE process. To this end, we distill definitions of explainability into an own suggestion that is useful for software and requirements engineering. We use this definition as a starting point to create a model that represents the impacts of explainability across different quality dimensions. Finally, we construct a knowledge catalogue of explainability and its impacts that is framed along these dimensions.

978-1-6654-2856-9/21/$31.00 ©2021 IEEE

DOI 10.1109/RE51729.2021.00025

• II. Background and Related Work

Chung et al. [16] explain the importance of conceptual models and knowledge catalogues as resources for the use and re-use of knowledge during system development. Models and catalogues can compile either abstract or concrete knowledge. At a more abstract level, such artifacts can compile knowledge about different NFRs and their interrelationships with other quality aspects. Likewise, models and catalogues can also compile more concrete knowledge, such as about methods and techniques in the field that can be used to operationalize a given NFR. The knowledge required to develop such artifacts is typically derived from literature, previous experiences, and domain expertise. By making this knowledge available in a single framework, developers can draw on know-how beyond their own fields and use this knowledge to meet the needs of a particular project. Essentially, software engineers can use models and knowledge catalogues to facilitate the software design process.

Some researchers developed catalogues for specific domains based on the premise of the NFR framework. Serrano and Serrano [21] developed a catalogue specifically for the ubiquitous, pervasive, and mobile computing domain. Torres and Martins [23] propose the use of NFR catalogues in the construction of RFID middleware applications to alleviate the challenges of NFR elicitation in autonomous systems. They argue that the use of catalogues can reduce or even eliminate possible faults in the identification of functional and non-functional requirements. Carvalho et al. [24] propose a catalogue for invisibility requirements focused on the domain of ubiquitous computing applications. They emphasize the importance of software engineers understanding the relationships between requirements in order to select appropriate strategies to satisfy invisibility and traditional NFRs. Furthermore, they discovered that invisibility may impact other essential NFRs for the domain, such as usability, security and reliability.

On a general level, Mairiza et al. [14] conducted a literature review to identify conflicts among existing NFRs. They constructed a catalogue to synthesize the results and suggest that it can assist software developers in identifying, analyzing, and resolving conflicts between NFRs. Carvalho et al. [22] identified 102 NFR catalogues in the literature after conducting a systematic mapping study. They found that the most frequently cited NFRs were performance, security, usability, and reliability. Furthermore, they found that the catalogues are represented in different ways, such as softgoal interdependency graphs, matrices, and tables. The existence of so many catalogues illustrates their importance for RE and software design. Although these catalogues present knowledge about 86 different NFRs, none of them addresses explainability.

Since explainability has rapidly expanded as a research field in the last years, publications about this topic have become quite numerous, and it is hard to keep track of the terms, methods, and results that came up. For this reason, there have been numerous SLRs presenting overviews concerning certain aspects (e.g., used methods or definitions) of explainability research. Many of these reviews focus on a specific community or application domain. For instance, [25] focuses on explainability of recommender systems, [26] on explainability of robots and human-robot interaction, [27] on the human-computer interaction (HCI) domain, and [28] on biomedical and malware classification. Another focus of these reviews is to demarcate different, but related terms often used in explainability research (e.g., in [4] and [29]). For instance, the terms "explainablilty" and "interpretability" are sometimes used as synonyms and sometimes not.

Our review differs from others in the following ways. First, many other reviews do not have an interdisciplinary focus. Even if they do not focus on a specific community, they rarely incorporate views on explainability outside of computer science. Second, quality aspects are the pivotal focus of our work. To the best of our knowledge, only a few reviews explicitly include NFRs or quality aspects (most notably [25] and [30]). Finally, in contrast to preceding reviews, we do not only consider positive impacts of explainability on other quality aspects, but we also take negative ones into account.

• III. Research Goal and Design

We frame our study into the following three RQs:

RQ1: What is a useful definition of explainability for the domains of software and requirements engineering?

RQ2: What are the quality aspects impacted by explainability in a system context?

RQ3: How does explainability impact these quality aspects?

Since other disciplines have a long history working on explainability, their insights should prove valuable for software engineering and enable us to delineate the scope of the term explainability for this area. Accordingly, RQ1 focuses on harnessing the work of other sciences in the field of explainability to compile a definition that is useful for the area of software and requirements engineering.

RQ2 focuses on providing an overview of the quality aspects that may be impacted by explainability. Similar to the work of Leite and Capelli [31], who investigated the interaction between transparency and other qualities, our goal is to offer an overview for explainability and its impact on other quality aspects within a system.

198

With RQ3 we want to assess what kind of impacts explainability has on other quality aspects. More specifically, our goal is to analyze the polarity of these impacts: whether they are positive or negative. To answer RQ2 and RQ3, we build a model and a catalogue that compiles knowledge about the impacts of explainability on other quality aspects.

Data Collection

SLR, Coding, £    & Analysis

Data Validation

Knowledge Structuring

Workshops 9

Model

& Catalogue

Fig. 1. Overview of the research design

An overview of our research design is shown in Fig. 1. Our research consisted of a multi-method approach that combined two qualitative methods to achieve higher reliability of our data. The first method focuses on systematic data collection and qualitative data analysis. For the data collection, we conducted an interdisciplinary SLR that resulted in a total of 229 papers. We coded the gathered data by using an open coding approach [32]. As a next step, we analyzed the resulting codes for definitions of explainability (RQ1), for relationships between explainability and other quality aspects (RQ2), and for information about the polarity of these relationships (RQ3). To validate and complement our findings, we employed a second qualitative method: two workshops with experts. Finally, we framed the obtained knowledge in a model by structuring and grouping the quality aspects impacted by explainability along four dimensions and developed our catalogue based on it.

The focus of this paper is on the results obtained through the qualitative research we conducted. For details on the SLR, especially the inclusion/exclusion criteria and the complete list of the papers analyzed through our literature review, please refer to our supplementary material [1]. A more detailed description of the workshops can also be found there. Further results and details about the methodology employed in our literature review will be addressed in a future publication.

• A. Data Collection and Analysis

1) Systematic Literature Review: We followed guidelines from Kitchenham et al. [33], and Wohlin [34] when conducting our SLR. The search strategy for our SLR consisted of a manual search followed by a snowballing process.

The manual search was performed independently by the authors of this paper and resulted in 104 papers. We used Fleiss’ Kappa statistics [35] to assess the reliability of the selection process. The calculated value of k = 0.81 showed an almost perfect agreement [36]. After the manual search, we performed snowballing to complement the search results. Our snowballing procedure included forward and backward snowballing [34] and resulted in additional 125 papers. Overall, our SLR yielded a total of 229 papers. The snowballing step was also independently conducted by the authors. The calculated value of k = 0.87 also shows an almost perfect agreement.

This literature review process is partially based on a grounded theory (GT) approach for literature reviews proposed by Wolfswinkel et al. [37]. The goal of using this approach to reviewing the literature is to reach a detailed and relevant analysis of a topic, following some of the principles of GT. According to [37], a literature review is never complete but at most saturated. This saturation is achieved when no new concepts or categories arise from the data. We followed this approach to decide when to conclude our snowballing process.

2) Coding and Analysis: We followed an open-coding approach [32] for the qualitative analysis of the papers we found during our search. This approach consists of up to three consecutive coding cycles. For our first coding cycle, we applied Initial Coding [38] to preserve the views and perspectives of the authors in the code. In the second coding cycle, we clustered the initial codes based on similarities, using Pattern Coding [39]. This allowed us to group the data from the first coding cycle into categories. Next, we discussed these categories until we reached an agreement on whether they adequately reflected the meaning behind the codes. These categories allowed us to structure the data for better analysis and to identify similarities.

For RQ2 and RQ3, we conducted a third coding cycle to further classify the categories into quality aspects. We applied Protocol Coding [40] as a procedural coding method in this cycle. For this method, we used a pre-established list of NFRs from Chung et al. [16]. If any correspondence between a category and an NFR was f