Abstract
Medical education programs must deliver valued results that stakeholders expect in return for their funding investments. In the past, healthcare organizations accepted reports about test results and participant perceptions of the program as adequate evidence of course outcomes. Today, program funders expect evaluations that provide evidence that medical education programs improve organizational excellence measures to justify ongoing funding. This chapter will explain four of the five elements required of a proven, comprehensive evaluation system. This five-element system is necessary to provide the desired organizational excellence evidence that medical educators can adopt to address the needs of stakeholders at different levels of an organization. Specifically, this chapter will overview an evaluation framework, a process model, and guiding principles that are crucial elements of this methodology. The chapter ends with a case study that shows how a medical education team used this measurement and evaluation methodology to plan how they would design and evaluate a medical education program requested by executives to solve an ICU central line infection problem.
TopIntroduction
A well-known quote attributed to Yogi Berra states, “When you come to a fork in the road, take it.” Medical education is standing at a fork in the road and must be ready to take it. Medical educators must be prepared to go left to create a traditional classroom setting or go right to use learning technologies such as simulators, distance learning, mobile learning, or even virtual/augmented/mixed reality platforms. Today, this is more complicated. Even the forks have forks. Yogi was ahead of his time.
Medical educators place great emphasis on creating sound instructional programs that achieve the desired learning outcomes. These rigorous, theory-based programs are essential. However, they lead to other forks in the road. Should medical educators take the familiar path where the struggle to show value is more faith-based (i.e., have faith in us and what we do delivers value) than based on credible measures that resonate with stakeholders? Or should they take the other path where the challenge is to show the evidence-based value stakeholders want to see—the impact and return on investment (ROI) they expect?
The objectives of this chapter are:
- 1.
Explain why medical educators must show the value of their programs using measures that matter to different stakeholders.
- 2.
List the five elements required of a comprehensive evaluation system required to help stakeholders see and show the value of what medical educators deliver.
- 3.
Describe the evaluation framework necessary to establish a chain-of-impact to measure different types of values that matter to stakeholders.
- 4.
Explain the four-phase, 12-step process model used to collect and analyze the data to show the value of a medical education program.
- 5.
Recall three common techniques used to isolate the effects of a medical education program.
- 6.
List the 12 guiding principles used to implement the evaluation process model credibly.
- 7.
Recall a case study showing how a medical education team used the ROI Methodology® to plan the design and evaluation of an education program requested by executives to solve an ICU central line infection problem.
TopBackground
Today, evaluation is about showing value that matters to key stakeholders. The challenge is “value” is in the eye of the beholder. Healthcare delivery executives and functional managers define value differently. Medical educators and healthcare providers also have unique definitions of value. When evaluating medical education programs, medical educators must have an evaluation methodology that shows the value of these programs in measures that matter to every stakeholder.
Medical educators value measures that indicate individuals or teams learned what was expected and had a favorable impression of the learning experience. Other stakeholders want to know if workplace performance improved. Executives want to know if the improved workplace performance subsequently improved the business measures they expect in return for the funding investment. This array of expectations requires a comprehensive evaluation framework showing a value continuum. This value continuum must balance six types of measures that do not reflect a single point-of-view or point-in-time perspective. These measures must provide financial and non-financial, quantitative and qualitative, tactical and strategic, and credible data to the stakeholders (Buzachero, Phillips, Phillips, & Phillips, 2013).
Even more, a comprehensive evaluation methodology must also answer the question, “How do you know the impact you are claiming is the result of your program and not something else?” The evaluation study must give impact credit where it is due and to claim credit with evidence (J. J. Phillips & Phillips, 2009). Finally, the medical educator must also know how to report the results of program evaluations in a way that executives, managers, and educators can take action to sustain and improve the learning program.
Key Terms in this Chapter
Evaluation Guiding Principles: Provide guidance to ensure consistency in decisions made at each step of the process model to protect the credibility of evaluation results with all stakeholders and to allow a credible comparison between different evaluation studies.
Evaluation Chain of Impact: A logic model that reflects how success or failure to address a need at a lower evaluation level affects the success or failure of the remaining evaluation levels to achieve program objectives. The evaluation chain of impact tells the complete story of the medical education programs success and shortcomings to achieve needs-driven program objectives.
ROI Methodology® Process Model: A comprehensive measurement and evaluation process that generates six types of measures—reaction and planned action, learning, application and implementation, impact, return on investment, and intangibles. It is a balanced measurement approach that includes a step to isolate the effects of a program, project, or solution.
ROI Institute: Founded in 1992 to help organizations evaluate the success of projects and programs, including measuring the financial return on investment (ROI). ROI Institute provides workshops, consulting, coaching, hosting briefings and presentations, researching, and benchmarking. It operates through a network of partners and associates in the U.S. and in over 70 countries with the help of over 100 ROI consultants.
Evaluation: There are three evaluation phases. A formative evaluation occurs during the design and development of an intervention where the focus is to improve its input-process-output before it is implemented. A summative evaluation occurs during the intervention to determine how well the program did to close the immediate reaction, learning, and application gaps defined by program objectives for each of these program evaluation levels. A confirmative evaluation occurs after the intervention to determine how well the program did to close the lagging impact results gap and achieve the desired ROI efficiency objective. This last evaluation phase also recommends improvements to the program evaluated so decision-makers can act to address program shortcomings and improve program processes.
Assessment: Defines the needs that justify funding a medical education program. A need is a gap between as-is and should-be results. An assessment also prioritizes those gaps based on their consequences and costs to the organization. An assessment differs from an analysis, which determines the causes of the gaps and identifies feasible solutions to address the identified program needs.
Evaluation Levels: Categories of data that reflect both quantitative and qualitative, as well as financial and non-financial, measures that are collected at different times to make evidence-based decisions about the effectiveness and efficiencies of the medical education program.
Return on Investment: A financial metric that compares the monetary benefits of an investment against the program costs of the investment. It is considered the ultimate economic measure of program success used by executives and senior leaders to determine the value of a program, project, or initiative.
Evaluation Impact Isolation: Provides evidence that allows the medical education program to take credit for impact measure improvements and to give credit elsewhere, where appropriate, since other variables can improve the same impact measures. Without isolation, an evaluation study is not valid when reporting the program's impact results.