Abstract
Methods to systematically document the health professionals’ perspective on relevance, cognitive impacts, use, and health outcomes of information items delivered by or retrieved in electronic knowledge resources (EKRs) may enhance evaluation of these resources, continuing education, and two-way knowledge ex change between information users and providers. The present chapter aims to overview seven years of research and development pioneering a comprehensive and systematic information assessment method (IAM), which has been validated for information delivery networks, information retrieval technology, and decision support systems. Using qualitative, quantitative, and mixed methods studies, we will support the feasibility, content validity, and construct validity of the IAM checklist combined with a computerized ecological momentary assessment technique for efficiently evaluating ‘relevance-impact-use-outcomes’ of information items derived from all these types of EKR. We will then present the current implementation of IAM 2008 for assessing e-therapeutics, an electronic textbook that provides updated therapeutic options for Canadian health professionals.
TopIntroduction
Developing new useful methods to evaluate Electronic Knowledge Resources (EKRs) for health professionals is a challenge. Although technology improves access to information, EKRs do not integrate self-assessment tools for systematically evaluating information items. There are numerous questionnaires to evaluate users’ satisfaction with EKRs, and there are few studies on the global use of research-based information in health sciences. However, to our knowledge, there are no published studies outside of our work that concomitantly examine the relevance, impact and use of information derived from EKRs in a systematic and comprehensive manner. Building on seven years of research and development, the present chapter proposes a unique Information Assessment Method (IAM) that may be considered as systematic and comprehensive.
IAM is important since health organizations are ‘knowledge-intensive’ firms relying heavily on information and communication technology, and specifically on EKRs. IAM is a unique validated method to systematically assess information items from the perspective of health professionals. The chapter will describe the development of this method and its potential impact for continuing education and knowledge exchange. We will present current research that shows how IAM may (1) develop continuing education for health professionals, specifically by operationalizing the concept of reflective learning, and (2) enable two-way knowledge exchange between information providers and health professionals, and so integrate knowledge translation into organizational routines.
The literature on research utilization started in the 1970s in social sciences, and the development of knowledge translation activities in health sciences has become increasingly popular in Canada. “Knowledge translation, commercialization and industry collaboration are all aimed at engaging stakeholder communities in the funding and translation of research for effective and innovative changes in health policy, practice or products” (http://www.cihr-irsc.gc.ca/e/29529.html). However, health research on knowledge translation is underdeveloped (Graham et al., 2006): theories are needed (Grimshaw et al., 2005), and few empirical studies scrutinize knowledge translation processes and outcomes (Estabrooks et al., 2003a). Four problems hinder this development: (1) the difficulty in identifying research-based information units for evaluation purposes; (2) the lack of studies beyond basic notions on the utilization of information; (3) the use of questionnaires with unknown validity; and (4) the absence of consensus on basic concepts (Estabrooks et al., 2003a; Graham et al., 2006; Grimshaw et al., 2005; Rich, 1997).
IAM addresses the first three problems. (1) Push and pull technology permits the delivery and retrieval of well-defined information units each with a unique identifier that may be assessed using IAM (e.g., a webpage). (2) IAM is based on a generic conceptual framework derived from information science, which integrates the acquisition, cognition and application of information. (3) The validity of IAM has been supported by seven years of research and development, and IAM has been used on different platforms and for three types of EKRs: email, information retrieval technology and decision support systems. Using qualitative, quantitative and mixed methods studies, our previous work supported the feasibility, content validity and construct validity of the IAM checklist combined with a Computerized Ecological Momentary Assessment technique for efficiently evaluating information items derived from all these types of resources.
Key Terms in this Chapter
Information Processes, Items, and Hits: In line with a generic theoretical model of information sciences, health professionals may receive or retrieve information (acquisition), integrate it (cognition), and use it (application). We operationalize this model as follows. An information item is defined as a passage of text in digital form with a unique identifier (e.g., a web page containing an abstract or a synopsis of a single study or a literature review with web-link to the original full-text publication). Such item is any discrete unit of information derived from electronic knowledge resources in response to a user query. Information items explicit clinical knowledge in a digital form. An information hit occurs when a health professional accesses or “opens” an information item. For instance, research-based information items may be received (push) or retrieved (pull), but are not necessarily opened. Health professionals screen titles of information items, and open only those that may be relevant. The relevance, cognitive impact, use and expected health outcomes of these information hits can then be documented using IAM (Information Assessment Method).
Electronic Knowledge Resources: refer to at least three types of social action and corresponding information and communication technologies: communication (emailing channels), programmed decision-making (decision support systems) and non-programmed decision-making (information retrieval technology). Electronic knowledge resources provide items of information in response to queries entered by health professionals (e.g., web pages).
Computerized Ecological Momentary Assessment (CEMA): permits to self-report observations, behaviours and cognitive processes in context, and to track record this information or data over time. CEMA is also referred to as experience sampling methods. CEMA decreases recall bias as compared to usual methods where participants are asked to remember their observations, behaviours and cognitive processes at a later time, or try to summarize typical situations and responses.
Information Delivery Networks: (push technology) refer to computer-mediated communication. They correspond to services that send information to health professionals on a regular basis or when new information is available (e.g., email alerting services). In such networks, health professionals may also communicate and discuss information used or not used (i.e., explaining knowledge and experiences that help each other via organizational learning or communities of practice). They can exchange knowledge through virtual meetings, electronic conferencing, and online seminars.
Decision Support Systems: or expert systems correspond to clinical situations that may be represented by unambiguous algorithms and then computerized (programmed decision-making). They match individual patient data with research and reference information to provide patient-specific information and guidance. They consist typically of clinical decision rules and calculators, which rationalize medicine as clinical protocols do. They require clinicians to enter patient-related data into a computer to obtain risk estimates, probability of diagnosis, prognosis and treatment recommendations.
Reflective Learning: informs continuing education, but is not well operationalized in the educational literature. It is defined as cognitive processes and tasks associated with learning situations. In line with the literature on higher order thinking, five major cognitive processes of reflective learning are as follows: the description of and the inference from evidence, the selection of evidence in preference to others, and its verification and synthesis.
Information Retrieval Technology: (pull technology) is defined as databases and search engines that health professionals can use to retrieve general information on disease prevention, health promotion, diagnosis, treatment and prognosis when needed (non-programmed decision-making). Databases contain organized materials that may be structured specifically to support clinical practice (e.g., PubMed). These materials can be presented in the form of electronic text, multimedia, or interactive media.