Evaluation of Health Information Systems: Challenges and Approaches

Introduction

It is hard to imagine health care without modern information and communication technology (ICT). It is evident that the use of modern information technology offers tremendous opportunities to reduce clinical errors, to support health care professionals and to increase the efficiency of care, and even to improve the quality of patient care (Institute of Medicine, 2001).

However, there are also hazards associated with ICT in health care: modern information systems are costly, their failures may cause negative effects on patients and staff, and possibly, when inappropriately designed, they may result in spending more time with the computer than with the patient. This all could have a negative impact on the efficiency of patient care. Therefore, a rigorous evaluation of IT in health care is recommended (Rigby, 2001) and of great importance for decision makers and users (Kaplan & Shaw, 2002). Evaluation can be defined as the decisive assessment of defined objects, based on a set of criteria, to solve a given problem (Ammenwerth et al., 2004).

The term ICT refers to technologies as such. Whether the use of these technologies is successful depends not only on the quality of the technological artefacts but also on the actors, i.e. the people involved in information processing and the organisational environment in which they are employed. ICT embedded in the environment, including the actors, is often referred to as an Information System (IS) in a sociotechnical sense. (Winter et al., 2001), (Berg, Aarts, & van der Lei, 2003).

Many different questions can lead the evaluation of information technology. Within evaluation research, two main (and often rather distinct) traditions can be found: the objectivist (positivistic) and the subjectivistic tradition (Friedman & Wyatt, 1997), which are related to the dominant use of either quantitative or qualitative methods (for details, see chapter 13).

Despite a large amount of published evaluation studies – e.g., (van der Loo, 1995) found over 1.500 citations on evaluation of healthcare IT between 1967 and 1995, and (Ammenwerth & de Keizer, 2004) found 1.035 studies between 1982 and 2002 - many authors report problems during evaluation. One of the main problems frequently discussed is the adequate choice of evaluation methods. While objectivistic researchers tend to concentrate on quantitative methods, subjectivistic researchers mainly rely on qualitative methods. Sometimes, a mixture of methods is applied. For example, qualitative methods are used to prepare quantitative studies, or quantitative measurements are used to support qualitative argumentation. However, there is still usually one tradition which dominates typical evaluation studies, leading to a focus either on quantitative or qualitative methods.

Many researchers point to the fact that this domination of one method or tradition may not be useful, but that a real integration of various methods from both traditions can be much more helpful to get comprehensive answers to given research questions. The integration of the complementary methods (and even beyond this, of data sources, theories and investigators), is discussed under the term “triangulation”.

In this paper, we first want to review some of the underlying reasons which make evaluation of health care IT so difficult. We will structure the problems into three main problem areas: the complexity of the object of evaluation, the complexity of the evaluation project, and the motivation to perform evaluation. We will discuss means how to overcome the discussed problems.

As one more detailed example, we then discuss what benefits can be obtained from applying triangulation in an evaluation study. Based on the example of the evaluation of a nursing documentation system, we show how both the validation of results and the completeness of results can be supported by triangulation.