Explanations in Artificial Intelligence Decision Making: A User Acceptance Perspective

Introduction

Artificial Intelligence (AI) systems are moving out of the laboratory and into the real world at an extremely rapid pace. As they do, the decisions they make affect people’s lives in diverse domains such as medical treatments and diagnoses, hiring and promotions, loans and the interest rates borrowers pay, prison sentences and so on (Pasquale, 2015; Roshanov et al., 2013; Wexler, 2018). This adoption has produced a corresponding increase in ethical concerns about how these decisions are made, and for good reason, as these decisions can be inaccurate or unacceptable (Guidotti et al., 2018). Such concerns have led to calls for AI systems to explain their decisions (Campolo, Sanfilippo, Whittaker, & Crawford, 2017; Edwards & Veale, 2017; Guidotti et al., 2018; Molnar & Gill, 2018; Wachter, Mittelstadt, & Russell, 2018). In this context, this chapter provides testable principles to which such explanations should conform to be acceptable to users of AI systems; both to the people who use the AI systems to generate decisions and to those who are subject to the decisions. If users do not find AI decisions-making system explanations acceptable, they will either not use the systems to generate decisions or refuse to comply with, or protest those decisions (Guidotti et al., 2018)¹.

Because the acceptability of AI explanations to users is a nascent area of research, the authors did not perform a systematic review of the topic. Rather, the authors searched broadly for journal articles and grey literature that might be relevant to the issue, using keywords such as “ethic*”, “explanation*” and “artificial intelligence”. Backward snowballing (Wohlin, 2014) led to other relevant articles. A review of the articles resulted in the proposal of three principles to which AI decision-making systems (AIDMS) should conform to support user acceptance. The principles were formulated to be testable, that is, to allow experiments to reveal whether explanations that conform to these principles are preferred by users, and/or are sufficient for users to accept the corresponding decisions. These proposed principles are fairness, contestability, and competence.

Fairness requires the equitable treatment of various groups of people and reasonable processing of relevant data. Contestability refers to the ability of a person about whom a decision is made to contest that decision. Competence requires the AIDMS’ decision-making performance to meet a certain quality threshold. While these principles do not refer directly to AIDMS explanations, they do generate requirements for such explanations to respect the principles. To illustrate how the principles relate to explanations, the principles are coupled with real world cases identified through backward snowballing (Wohlin, 2014) of the articles and reports collected and through Google searches. Such case-based illustrations are a common educational approach in many fields (Johnson et al., 2012).

These three testable principles constitute this chapter’s primary contribution to the field. The authors make a secondary contribution through their discussion of the varying potential of different AIDMSs to adhere to the proposed principles, showing that not all so-called black box (Guidotti et al., 2018) AIDMSs are equally opaque.

Philosophical investigations of the nature of explanations (e.g. Tuomela, 1980) were not examined, neither were the ways in which people generate explanations themselves (e.g. Hilton, 1990). Psychological research on explanation was also not examined. On the whole, psychology has focused on how people explain other people’s behavior (Malle, 2011), the conditions that induce people to make cause and effect inferences (e.g. Subbotsky, 2004), the extent to which people’s explanations of events are consistent with scientific ones (Kubricht, Holyoak, & Lu, 2017), and the formation of mental models about real-word events (Battaglia, Hamrick, & Tenenbaum, 2013). It is not that that this research is irrelevant to the issue, but it is just not as directly applicable as material that is specifically about AIDMS explanations and concerns.