Spreadsheets are commonly used and commonly flawed, but it is not clear how often spreadsheet errors lead to bad decisions. We interviewed 45 executives and senior managers/analysts in the private, public, and non-profit sectors about their experiences with spreadsheet quality control and with errors affecting decision making. Almost all of them said spreadsheet errors are common. Quality control was usually informal and applied to the analysis and/or decision, not just the spreadsheet per se. Most respondents could cite instances of errors directly leading to bad decisions, but opinions differ as to whether the consequences of spreadsheet errors are severe. Some thought any big errors would be so obvious as to be caught by even informal review. Others suggest that spreadsheets inform but do not make decisions, so errors do not necessarily lead one for one to bad decisions. Still, many respondents believed spreadsheet errors were a significant problem and that more formal spreadsheet quality control could be beneficial.
TopIntroduction
Spreadsheets are used in diverse domains by decision makers at all levels (Gerson, Chien, & Raval, 1992; Chan and Storey, 1996; Seal, Przasnyski, & Leon, 2000; Croll, 2005); the entire July-August, 2008 issue of the journal Interfaces is devoted to spreadsheet modeling success stories. However, laboratory studies and field audits consistently find that a large proportion of spreadsheets contain errors. In a dozen studies reviewed by Kruck, Maher, & Barkhi (2003), the average proportion of spreadsheets with errors was 46%. Panko’s (2000a, 2005) synthesis of spreadsheet audits published since 1995 suggested a rate of 94%. Powell, Baker, & Lawson (2007a, 2008b) critique past work and greatly advance methods of defining and measuring spreadsheet errors but at the end of the day reach the same overall error rate of 94%.
Hence, one might expect that (1) spreadsheet errors frequently lead to poor decisions and (2) organizations would invest heavily in quality control procedures governing spreadsheet creation and use. We investigated both hypotheses through 45 semi-structured interviews with executives and senior managers / analysts in the public, nonprofit, and private sectors. Field interviews raise fewer concerns about external validity than do laboratory studies, and they focus on the overall decision making process, not just the spreadsheet artifact as in audit studies. However, our approach has two important limitations. First, the respondents are a convenience sample. Second, self-report can be flawed, whether through imperfect memories, self-serving bias, conscious deception, and/or limited self-awareness. Given these limitations, we focus on broad qualitative conclusions. In brief, we found that most respondents could describe instances in which spreadsheet errors contributed to poor decisions, some with substantial consequences, yet few reported that their organization employs quality control procedures specific to spreadsheet analysis.
The literature on spreadsheet errors in general is large (see Panko, 2000b and Powell, Baker, & Lawson, 2008a for reviews), but much less has been written on these specific questions. Regarding the frequency with which spreadsheet errors lead to bad decisions, the European Spreadsheet Research Interest Group (EUSPRIG) maintains a webpage of news stories reporting the consequences of spreadsheet errors (http://www.eusprig.org/stories.htm). However, spreadsheets are used by so many organizations that even if only a small proportion were hurt badly by spreadsheet errors, there could still be scores of examples. We started with a population of individuals and organizations for which we had no a priori reason to think spreadsheet errors were a particular problem. This approach has been taken by others (e.g., Cragg and King, 1993) to explore the prevalence of defective spreadsheets, but like Powell, Baker, & Lawson (2007b), we shift the focus to assessing the impact of those spreadsheet errors.
A considerable corpus on controlling spreadsheet errors concerns what organizations should do. Classic recommendations lean toward application of good software engineering principles (Mather, 1999; Janvrin and Morrison, 2000; Rajalingham, Chadwick, & Knight, 2000; Grossman and Özlük, 2004) or formal theories (Isakowitz, Schocken, & Lucas, 1995). Kruck and Sheetz (2001) combed practitioner literature for practical axioms validated by empirical results, supporting aspects of the spreadsheet lifecycle theory (e.g., include planning / design and testing/debugging stages) and recommendations to decrease formula complexity.
There is also some literature describing what organizations actually do. Notably, Finlay and Wilson (2000) surveyed 10 academics and 10 practitioners on the factors influencing spreadsheet validation. Those most commonly mentioned were (a) aspects of the decision and (b) aspects of the spreadsheet underlying the decision context. However, Grossman (2002) argues that it would be valuable to have greater knowledge of what error control methods are currently used.