Article Preview
TopIntroduction
Nowadays, spreadsheet-based systems (or simply spreadsheets) have become an indispensable tool in various business areas such as accounting and financial reporting, asset recording, production scheduling, and engineering design (Lu et al., 1991; McDaid & Rust, 2009). Spreadsheets are often used to generate important information for managers and executives in making strategic decisions (Caulkins et al., 2007; Kruck et al., 2003).
Although spreadsheet applications are extremely popular, numerous spreadsheet failures [1] have been reported in the literature (Bishop & McDaid, 2011; Leon et al., 2015; Morrison et al., 2002; Panko, 1998, 1999, 2007; Panko & Aurigemma, 2010). Some studies further substantiate the widespread growth of faulty spreadsheets. For example, Panko & Ordway (2005) found that nearly all (94%) of the spreadsheets contained faults. A possible reason contributing to the large number of faulty spreadsheets is that their development has shifted from IT professionals to “non-technical” departmental end users such as accounting or marketing staff (hereafter referred to as end-user programmers); many of the latter have little formal training in software development and testing (Powell et al., 2008, 2009). According to the European Spreadsheet Risks Interest Group (EuSpRIG), faulty spreadsheets could result in various business risks including: (a) loss in revenue, profit, cash, assets, and tax; (b) mispricing and poor decision-making; and (c) financial failure.
In view of the above risks, several systematic techniques have been developed for dynamic testing (that is, testing involving software execution) of spreadsheets. These techniques include the constraint-based spreadsheet testing method (Abraham & Erwig, 2006), the “What You See Is What You Test (WYSIWYT)” methodology (Fisher et al., 2006), error trapping (ET) (Jain, 2010; Powell & Baker, 2009, pp.115−116), and metamorphic testing (MT) (Chen et al., 2003; Liu et al., 2014;Poon et al., 2014).
Among these techniques, the constraint-based spreadsheet testing method requires formal training in software engineering, and the WYSIWYT methodology requires some technical knowledge of data-flow adequacy criteria (Jee et al., 2009) and coverage monitoring (Vilkomir et al., 2003). These requirements pose difficulties to end-user programmers who often do not possess such technical knowledge. On the other hand, ET and MT are more applicable to end-user programmers because both require less technical knowledge for their use [2]. Since this paper is related to dynamic spreadsheet testing from the end-user programmer’s perspective, ET and MT are the main focus.
In the course of our research into spreadsheet testing, we observe that many people confuse between ET and MT, because they are similar in some aspects. Thus, this paper aims to help readers understand the two concepts. It also investigates their failure detection effectiveness, and determines whether they are complements or substitutes. The paper first analytically compares the two techniques, specifically focusing on their similarities, differences, and relationship. The analytical comparison is followed by a discussion of an experiment, which investigates their effectiveness in detecting spreadsheet failures. Thereafter, a practical guidance on spreadsheet testing with the use of ET and MT are provided.