Error Trapping and Metamorphic Testing for Spreadsheet Failure Detection

Error Trapping and Metamorphic Testing for Spreadsheet Failure Detection

Pak-Lok Poon (RMIT University, Melbourne, Australia), Huai Liu (RMIT University, Melbourne, Australia) and Tsong Yueh Chen (Swinburne University of Technology, Melbourne, Australia)
Copyright: © 2017 |Pages: 18
DOI: 10.4018/JOEUC.2017040102
OnDemand PDF Download:
No Current Special Offers


This study deepens the research on error trapping (ET) and metamorphic testing (MT) for detecting spreadsheet failures. It is observed that some spreadsheet developers and testers are confused between ET and MT, because the two techniques are similar to each other in some aspects. Inspired by the observation, this paper first outlines the main concepts of ET and MT using several examples for illustration. This is followed by discussing an experiment with a view to investigating and comparing the failure detection capabilities of the two techniques. The results of the experiment indicate that neither technique is sufficient in spreadsheet testing. Rather, ET and MT are complementary and they should be used together in spreadsheet testing whenever possible.
Article Preview


Nowadays, spreadsheet-based systems (or simply spreadsheets) have become an indispensable tool in various business areas such as accounting and financial reporting, asset recording, production scheduling, and engineering design (Lu et al., 1991; McDaid & Rust, 2009). Spreadsheets are often used to generate important information for managers and executives in making strategic decisions (Caulkins et al., 2007; Kruck et al., 2003).

Although spreadsheet applications are extremely popular, numerous spreadsheet failures [1] have been reported in the literature (Bishop & McDaid, 2011; Leon et al., 2015; Morrison et al., 2002; Panko, 1998, 1999, 2007; Panko & Aurigemma, 2010). Some studies further substantiate the widespread growth of faulty spreadsheets. For example, Panko & Ordway (2005) found that nearly all (94%) of the spreadsheets contained faults. A possible reason contributing to the large number of faulty spreadsheets is that their development has shifted from IT professionals to “non-technical” departmental end users such as accounting or marketing staff (hereafter referred to as end-user programmers); many of the latter have little formal training in software development and testing (Powell et al., 2008, 2009). According to the European Spreadsheet Risks Interest Group (EuSpRIG), faulty spreadsheets could result in various business risks including: (a) loss in revenue, profit, cash, assets, and tax; (b) mispricing and poor decision-making; and (c) financial failure.

In view of the above risks, several systematic techniques have been developed for dynamic testing (that is, testing involving software execution) of spreadsheets. These techniques include the constraint-based spreadsheet testing method (Abraham & Erwig, 2006), the “What You See Is What You Test (WYSIWYT)” methodology (Fisher et al., 2006), error trapping (ET) (Jain, 2010; Powell & Baker, 2009, pp.115−116), and metamorphic testing (MT) (Chen et al., 2003; Liu et al., 2014;Poon et al., 2014).

Among these techniques, the constraint-based spreadsheet testing method requires formal training in software engineering, and the WYSIWYT methodology requires some technical knowledge of data-flow adequacy criteria (Jee et al., 2009) and coverage monitoring (Vilkomir et al., 2003). These requirements pose difficulties to end-user programmers who often do not possess such technical knowledge. On the other hand, ET and MT are more applicable to end-user programmers because both require less technical knowledge for their use [2]. Since this paper is related to dynamic spreadsheet testing from the end-user programmer’s perspective, ET and MT are the main focus.

In the course of our research into spreadsheet testing, we observe that many people confuse between ET and MT, because they are similar in some aspects. Thus, this paper aims to help readers understand the two concepts. It also investigates their failure detection effectiveness, and determines whether they are complements or substitutes. The paper first analytically compares the two techniques, specifically focusing on their similarities, differences, and relationship. The analytical comparison is followed by a discussion of an experiment, which investigates their effectiveness in detecting spreadsheet failures. Thereafter, a practical guidance on spreadsheet testing with the use of ET and MT are provided.

Complete Article List

Search this Journal:
Volume 34: 10 Issues (2022)
Volume 33: 6 Issues (2021)
Volume 32: 4 Issues (2020)
Volume 31: 4 Issues (2019)
Volume 30: 4 Issues (2018)
Volume 29: 4 Issues (2017)
Volume 28: 4 Issues (2016)
Volume 27: 4 Issues (2015)
Volume 26: 4 Issues (2014)
Volume 25: 4 Issues (2013)
Volume 24: 4 Issues (2012)
Volume 23: 4 Issues (2011)
Volume 22: 4 Issues (2010)
Volume 21: 4 Issues (2009)
Volume 20: 4 Issues (2008)
Volume 19: 4 Issues (2007)
Volume 18: 4 Issues (2006)
Volume 17: 4 Issues (2005)
Volume 16: 4 Issues (2004)
Volume 15: 4 Issues (2003)
Volume 14: 4 Issues (2002)
Volume 13: 4 Issues (2001)
Volume 12: 4 Issues (2000)
Volume 11: 4 Issues (1999)
Volume 10: 4 Issues (1998)
Volume 9: 4 Issues (1997)
Volume 8: 4 Issues (1996)
Volume 7: 4 Issues (1995)
Volume 6: 4 Issues (1994)
Volume 5: 4 Issues (1993)
Volume 4: 4 Issues (1992)
Volume 3: 4 Issues (1991)
Volume 2: 4 Issues (1990)
Volume 1: 3 Issues (1989)
View Complete Journal Contents Listing