Abstract
This article discusses approaches for evaluating distance education activities. It comprises a framework for evaluation that is based on widely adopted approaches to educational evaluation and which can be used for evaluating other educational activities as well. The critical components of this framework are threefold: various phases in the evaluation process, the main focus of each phase, and most appropriate strategies for gathering data in each phase. The use of a framework such as this will ensure that the evaluation process is systematic and also thorough. The discussion here extends earlier discussions of the topic by this author in two other publications. These are in a chapter titled “Designing and evaluating instruction for e-learning”, that is published in a book edited by Patricia Rogers “Designing Instruction for Technology-Enhanced Learning” (see Rogers, 2002), and in a chapter titled “Evaluating the impacts of e-learning” in the book “E-learning: A Guidebook of Principles, Procedures and Practices” (see Naidu, 2006).
TopBackground: The Evaluation Process
The term evaluation is sometimes used to refer to the process of assessing student achievement or performance. For our purposes here, however, we see “evaluation of the impacts of an educational program or project” as different from the “assessment of its learning outcomes” for students. Evaluation, as seen here, refers to the gathering and observation of a broad range of evidence on the impacts and effectiveness of a program, project or process. Assessment of its impacts on learning comprises the examination of the performance of learners which can contribute to the evaluation of the overall impact of the program or project.
The evaluation process comprises gathering of data on the use, worth, and impact of a program, project, or process in relation to its intended outcomes. Systematic gathering of this information is crucial to the successful development and implementation of any program or project. The evaluation process comprises front-end analysis, formative, summative, and monitoring or integrative evaluation (see Kirkpatrick, 1994; Naidu, 2002, 2005, 2006; Reeves, 1999).
Table 1 offers a framework for evaluating distance education activities or, for that matter, any other such educational activity. It draws its processes from widely adopted approaches to educational evaluation and comprises various phases of the evaluation process, their foci, as well as appropriate strategies for gathering relevant data (see Patton, 1988, 1990; Reeves, 1997; Shulman, 1988).
Table 1. A framework for evaluating distance education
| Goals: It is most important to be very clear about this, as this will determine what is done and how it is done. |
| Phases | Focus | Some strategies and resources for gathering data |
| Front-end analysis | Context analysis
User and user needs analysis | • Background documentation including policy documents
• Surveys of potential students, and other stakeholders, their backgrounds, needs, aspirations and expectations. |
| Formative evaluation | Design evaluation involves ascertaining the match between the goals of the program or project and its design architecture.
Prototype evaluation involves ascertaining from small group of users how the program matches predefined design criteria.
Pilot implementation involves gathering data from a small sample of users in a trial implementation of the project or program. | • Expert analysis
• Direct observation
• Analysis of user interactions and their products
• User’s self-reporting which includes feedback gained through questionnaires, checklists, video-stimulated recall/teach-back/discussion, and focus group interviews. |
| Summative evaluation | Impact evaluation examines the effects of the program or project on identified goals and outcomes. | • Surveys
• Focus group interviews.
• Clinical interviews with individual users. |
| Monitoring and integrative evaluation | Integration evaluation examines the extent to which the project or program is forming an integral part of the whole infrastructure. | • Survey of patterns of use and user satisfaction.
• Cost-benefit analysis |
Key Terms in this Chapter
Summative Evaluation: This is the process of gathering data from users following a full-scale implementation of a program or project.
Front-End Analysis: This is the process of collecting information in preparation for the design, development, and implementation of a program or project.
Formative Evaluation: This is the process of gathering data from a sample of users in order to inform the design and development process.
Evaluation: This is the process of gathering and observing evidence in order to gauge the impacts and effectiveness of a program or process.
Monitoring or Integrative Evaluation: This is the process of gathering data from users and other stakeholders on the uptake and adoption of a program or project.y