Due to the prevalent use of Information Systems (IS) in modern organisations, evaluation research in this field is becoming more and more important. In light of this, a set of rigorous methodologies were developed and used by IS researchers and practitioners to evaluate the increasingly complex IS implementation used. Moreover, different types of IS and different focusing perspectives of the evaluation require the selection and use of different evaluation approaches and methodologies. This chapter aims to identify, explore, investigate, and discuss the various key methodologies that can be used in IS evaluation from different perspectives, namely in nature (e.g. summative vs. formative evaluation) and in strategy (e.g. goal-based, goal-free, and criteria-based evaluation). Six case studies are also presented and discussed in this chapter to illustrate how the different IS evaluation methodologies can be applied in practices. The chapter concludes that evaluation methodologies should be selected depending on the nature of the IS and the specific goals and objectives of the evaluation. Nonetheless, it is also proposed that formative criteria-based evaluation and summative criteria-based evaluation are currently among the more widely used in IS research. The authors suggest that the combined used of one or more of these approaches can be applied at different stages of the IS life cycle in order to generate more rigorous and reliable evaluation outcomes. Moreover, results and outcomes of IS evaluation research will not just be useful in practically guiding actions to improve the current system, but can also be used to generate new knowledge and theory to be adopted by future IS research.
TopIntroduction
Evaluation research can be defined as a form of “disciplined inquiry” (Guba & Lincoln, 1981, p. 550), which “applies scientific procedures to the collection and analysis of information about the content, structure and outcomes of programmes, projects and planned interventions” (Clarke, 1999, p. 1). Both quantitative and qualitative methods, or even a mixed-methods approach, can be adopted in evaluation research. Clarke (1999, p. 2) highlights that the key to distinguish evaluation research from other forms of research is not the data collection methods being employed but the purpose for which these methods are used. In particular, it is important to note that the primary purpose or objective of evaluation research is not to explore new knowledge as other forms of research do (Clarke, 1999, p. 2). Rather, it aims at using current knowledge to assess and study the effects, effectiveness and outcomes of “some innovation, intervention, policy, practice or service” (Robson, 2002, p. 202), and then to inform decision making to guide practical actions (Clarke, 1999, p. 2; Lagsten & Goldkuhl, 2008).
This type of research started receiving substantial attention from academics since the 1960s (Robson, 2002, p. 203). Specifically, in the 1960s the US government invested a large amount of money in developing various new social programmes in education, income maintenance, housing, and health (Dart, et al., 1998). These vast investments raised the issue and need of evaluating the outcomes and impact of the developed social programmes, which subsequently turned into an interest in evaluation in Social Sciences research (Robson, 2002, p. 203; Dart, et al., 1998). In other words, evaluation research has its root in the field of Social Sciences.
In terms of Information Systems (IS) research, evaluation is particularly important. In fact, and according to the International Data Corporation 2007 report (IDC, 2008), the global software market reached US$229,946 million in 2007. This figure clearly indicates the prevalence and heavy investments of IS in modern organizations. However, and despite this apparent success in the IS market, failure rates of IS implementation and exploitation have been continuously high (Chen, et al., 2011; Peng & Nunes, 2009; Lycett & Giaglis 2000). For example, and according to a recent Standish Group Chaos Report (Standish Group, 2009), 44% of IS projects were considered as challenged and 24% were identified as a complete failure in 2008. Giving the large investment and high failure rate of IS implementation, evaluation is now recognized as an increasingly important task that can directly contribute to IS success (Ammenwerth, et al., 2003; Lycett & Giaglis, 2000).
In particular, Lycett and Giaglis (2000) argue that evaluation is very useful in predicting and assessing potential costs, benefits and risks associated with the development, implementation and use of IS, as well as assisting decision makers to take proper actions to mitigate the identified risks. Moreover, other IS researchers reinforce that in order to inform decision making and increase the possibility of IS success, evaluation should be carried out at different phases throughout the entire system’s lifecycle, from feasibility study, to system development, implementation, post-implementation and even system replacement (Willcocks & Lester, 1996; Smithson & Hirschheim, 1998; Seddon, et al., 2002).
However, and despite its importance in guaranteeing IS success, evaluation is never an easy and straightforward task (Cronholm & Goldkuhl, 2003a). In particular, there is a range of IS evaluation methodologies, each one having its own strengths and limitations. Moreover, different stages of the IS lifecycle are associated with different goals, changes and outcomes. As a result, the aims and focuses of evaluation at different stages will also vary. Faced with this diversity and complexity, practitioners and evaluators may often find it difficult to select which methodology is the most suitable one for evaluating a particular IS project or a particular stage of the project.