Evaluation of Health Information Systems: Challenges and Approaches

Evaluation of Health Information Systems: Challenges and Approaches

Elske Ammenwerth (University for Health Sciences, Medical Informatics and Technology (UMIT), Austria), Stefan Gräber (University Hospital of Saarland, Germany), Thomas Bürkle (University of Münster, Germany) and Carola Iller (University of Heidelberg, Germany)
Copyright: © 2009 |Pages: 16
DOI: 10.4018/978-1-60566-078-3.ch018
OnDemand PDF Download:


This chapter summarizes the problems and challenges which occur when health information systems are evaluated. The main problem areas presented are the complexity of the evaluation object, the complexity of an evaluation project, and the motivation for evaluation. Based on the analysis of those problem areas, the chapter then presents recommendations how to address them. In particularly, it discusses in more detail what benefits can be obtained from applying triangulation in evaluation studies. Based on the example of the evaluation of a nursing documentation system, it shows how both the validation of results and the completeness of results can be supported by triangulation. The authors hope to contribute to a better understanding of the peculiarities of evaluation.
Chapter Preview


It is hard to imagine health care without modern information and communication technology (ICT). It is evident that the use of modern information technology offers tremendous opportunities to reduce clinical errors, to support health care professionals and to increase the efficiency of care, and even to improve the quality of patient care (Institute of Medicine, 2001).

However, there are also hazards associated with ICT in health care: modern information systems are costly, their failures may cause negative effects on patients and staff, and possibly, when inappropriately designed, they may result in spending more time with the computer than with the patient. This all could have a negative impact on the efficiency of patient care. Therefore, a rigorous evaluation of IT in health care is recommended (Rigby, 2001) and of great importance for decision makers and users (Kaplan & Shaw, 2002). Evaluation can be defined as the decisive assessment of defined objects, based on a set of criteria, to solve a given problem (Ammenwerth et al., 2004).

The term ICT refers to technologies as such. Whether the use of these technologies is successful depends not only on the quality of the technological artefacts but also on the actors, i.e. the people involved in information processing and the organisational environment in which they are employed. ICT embedded in the environment, including the actors, is often referred to as an Information System (IS) in a sociotechnical sense. (Winter et al., 2001), (Berg, Aarts, & van der Lei, 2003).

Many different questions can lead the evaluation of information technology. Within evaluation research, two main (and often rather distinct) traditions can be found: the objectivist (positivistic) and the subjectivistic tradition (Friedman & Wyatt, 1997), which are related to the dominant use of either quantitative or qualitative methods (for details, see chapter 13).

Despite a large amount of published evaluation studies – e.g., (van der Loo, 1995) found over 1.500 citations on evaluation of healthcare IT between 1967 and 1995, and (Ammenwerth & de Keizer, 2004) found 1.035 studies between 1982 and 2002 - many authors report problems during evaluation. One of the main problems frequently discussed is the adequate choice of evaluation methods. While objectivistic researchers tend to concentrate on quantitative methods, subjectivistic researchers mainly rely on qualitative methods. Sometimes, a mixture of methods is applied. For example, qualitative methods are used to prepare quantitative studies, or quantitative measurements are used to support qualitative argumentation. However, there is still usually one tradition which dominates typical evaluation studies, leading to a focus either on quantitative or qualitative methods.

Many researchers point to the fact that this domination of one method or tradition may not be useful, but that a real integration of various methods from both traditions can be much more helpful to get comprehensive answers to given research questions. The integration of the complementary methods (and even beyond this, of data sources, theories and investigators), is discussed under the term “triangulation”.

In this paper, we first want to review some of the underlying reasons which make evaluation of health care IT so difficult. We will structure the problems into three main problem areas: the complexity of the object of evaluation, the complexity of the evaluation project, and the motivation to perform evaluation. We will discuss means how to overcome the discussed problems.

As one more detailed example, we then discuss what benefits can be obtained from applying triangulation in an evaluation study. Based on the example of the evaluation of a nursing documentation system, we show how both the validation of results and the completeness of results can be supported by triangulation.

Complete Chapter List

Search this Book:
Editorial Advisory Board
Table of Contents
Neil Pattison
Emma Parry, David Parry
Emma Parry, David Parry
Chapter 1
Peter Stone
Improving women’s health is a vital task for the world. The consequences of obstetric and gynecological disease are serious both for the women... Sample PDF
An Introduction to Women's Health and Informatics
Chapter 2
Premila Fade
Principlism (derived from common sense morality) is the most common theory used within the healthcare sphere. The elements of this theory are... Sample PDF
Women's Health Informatics: The Ethical and Legal Issues
Chapter 3
David Parry
Recording information about symptoms, observations, actions, and outcomes is a key task of health informatics. Standardization of records is vital... Sample PDF
Coding and Messaging Systems for Women's Health Informatics
Chapter 4
Gareth Parry
Women’s health in primary care is a large part of the generalist’s practice. Information technology (IT) is now an integral part of the generalist’s... Sample PDF
Women's Health Informatics in the Primary Care Setting
Chapter 5
Emma Parry
The seamless electronic health record is often hailed as the holy grail of health informatics. What is an electronic health record? This question is... Sample PDF
The Electronic Health Record to Support Women's Health
Chapter 6
Graham Parry
Information technology and communication systems have made imaging in women’s health easier at many levels. There are now many commercial systems on... Sample PDF
Imaging and Communication Systems in Obstetrics and Gynecology
Chapter 7
Emma Parry
Pregnancy is unique in medicine in providing a discrete event with a fixed end. It is well suited to data collection and statistical assessment.... Sample PDF
Statistical Measures in Maternity Care
Chapter 8
Kiran Massey, Tara Morris, Robert M. Liston
Our ultimate goal as obstetric and neonatal care providers is to optimize care for mothers and their babies. As such, we need to identify practices... Sample PDF
Building Knowledge in Maternal and Infant Care
Chapter 9
Malcolm Battin, David Knight, Carl Kuschel
Neonatal care is an extremely data-intensive activity. Physiological monitoring equipment is used extensively along with web-based information tools... Sample PDF
Informatics Applications in Neonatology
Chapter 10
Jenny Westgate
During pregnancy the fetus requires an adequate supply of oxygen and clearance of carbon dioxide which is a waste product of metabolism. In fetal... Sample PDF
Computerizing the Cardiotocogram (CTG)
Chapter 11
Liron Pantanowitz
Automation and emerging information technologies are being adopted by cytology laboratories around the world to augment Pap test screening and... Sample PDF
Computer Assisted Cervical Cytology
Chapter 12
Laurie Elit, Susan Bondy, Michael Fung-Kee-Fung, Prafull Ghatage, Tien Le, Barry Rosen, Bohdan Sadovy
Ovarian cancer affects 2,400 women annually in Canada with a case fatality ratio of 0.70. There are several practice guidelines that indicate women... Sample PDF
Informatics and Ovarian Cancer Care
Chapter 13
Jamila Abuidhail
Information and communication technologies include computers, telecommunication, digital networks, and television. Using informatics in healthcare... Sample PDF
Women's Health and Health Informatics: Perinatal Care Health Education
Chapter 14
Shona Kirtley
In an age where health professionals lead very busy working lives, electronic information sources provide ease of access to vast amounts of health... Sample PDF
Electronic Information Sources for Women's Health Knowledge for Professionals
Chapter 15
David Parry
Decision analysis techniques attempt to utilize mathematical data about outcomes and preferences to help people make optimal decisions. The... Sample PDF
Computerised Decision Support for Women's Health Informatics
Chapter 16
Michelle Brear
The influence of organizational factors on the success of informatics interventions in healthcare has been clearly demonstrated. This health... Sample PDF
Organizational Factors: Their Role in Health Informatics Implementation
Chapter 17
Josipa Kern
When things go well then often it is because they conform to standards (ISO, 2005). According to the Oxford Dictionary of Modern English, there is a... Sample PDF
Standardization in Health and Medical Informatics
Chapter 18
Elske Ammenwerth, Stefan Gräber, Thomas Bürkle, Carola Iller
This chapter summarizes the problems and challenges which occur when health information systems are evaluated. The main problem areas presented are... Sample PDF
Evaluation of Health Information Systems: Challenges and Approaches
Chapter 19
Pirkko Nykänen
eHealth refers to use of information and communication technologies to improve or enable health and healthcare. eHealth broadens the scope of health... Sample PDF
eHealth Systems, Their Use and Visions for the Future
Chapter 20
Nilmini Wickramasinghe, Santosh Misra, Arnold Jenkins, Douglas R. Vogel
Superior access, quality and value of healthcare services has become a national priority for healthcare to combat the exponentially increasing costs... Sample PDF
The Competitive Forces Facing E-Health
About the Contributors