Abstract
According to the Joint Committee on Standards for Educational Evaluation (1994), “Program evaluation is a systematic investigation of the worth of an ongoing or continuing distance education activity” (Simonson, 1997, p. 88). As such, this article addresses the issue of evaluating programs rather than courses. Although it is true that content, instructional design, and delivery greatly affect the quality of the program, course evaluation is a topic in and of itself. Frydenberg (2002) noted that program evaluation was frequently listed as a separate item in standards documentation: “While assessment of student achievement is normally described as part of instructional design and tied to specific course objectives, program evaluation is an activity that incorporates all the aspects of the e-learning experience” (p. 7). High-level aspects of course design are, however, built into program evaluation as you will see because it is impossible to evaluate an educational program without looking at courses.
TopResearch On Evaluating Online Programs
“The literature on e-learning program evaluation is naturally skimpy, since few fully developed programs have arrived at a stage where summative evaluation is possible” (Frydenberg, 2002, p. 11). This does not prevent, however, the ability to develop a formative and summative evaluation structure because the process for evaluating a distance learning program should be founded on the principles of evaluating any educational program. According to the Program Evaluation Standards established by 16 professional associations, sound evaluations (of educational programs, projects, and materials) in a variety of settings should have the following four attributes (ERIC, 1995):
- 1.
Utility: These standards are intended to ensure that an evaluation will serve the information needs of intended users. These include identifying all stakeholders, selecting a trustworthy and competent evaluator, collecting information that addresses pertinent questions and meets the stakeholders’ needs, clarifying value judgments, describing the program being evaluated clearly, distributing significant interim findings and evaluation reports in a timely fashion, and reporting findings in ways that encourage follow-through by stakeholders.
- 2.
Feasibility: These standards are intended to ensure that an evaluation will be realistic, prudent, diplomatic, and frugal. These include developing practical procedures to keep disruption to a minimum, anticipating stakeholders’ views to gain their cooperation and curtail bias or interference, and conducting an efficient evaluation that keeps costs down while producing information of value.
- 3.
Propriety: These standards are intended to ensure that an evaluation will be conducted legally, ethically, and with due regard for the welfare of those involved in the evaluation, as well as those affected by its results. These standards include designing evaluations that effectively serve the needs of the targeted participants, getting obligations agreed to in writing, respecting and protecting the rights and welfare of human subjects, respecting human dignity and worth in interactions with others, conducting a complete and fair examination that includes the program’s strengths and weaknesses, disclosing findings to all stakeholders, addressing any conflicts openly and honestly, and applying sound accountability procedures.
- 4.
Accuracy: These standards are intended to ensure that an evaluation will reveal and convey technically adequate information about the features that determine worth of merit of the program being evaluated. These include documenting the program clearly and accurately, examining the context in which the program exists, describing the purposes and procedures of the evaluation in enough detail so they can be identified and defensibly assessed, using valid and reliable information, reviewing quantitative and qualitative information systematically to find errors and effectively answer evaluation questions, justifying conclusions, avoiding distortion caused by personal feelings and bias, and evaluating the program formatively and summatively.
Key Terms in this Chapter
Process-Based Evaluation: A type of evaluation used to understand how a program works, as well as its strengths and weaknesses.
Outcomes-Based Evaluation: A type of evaluation used to determine the extent to which the program benefits its clients.
Goals-Based Evaluation: A type of evaluation used to determine the extent to which programs are achieving their overall, predetermined objectives.
Program Evaluation: “A systematic investigation of the worth of an ongoing or continuing distance education activity” (Simonson, 1997, p. 88).
Summative Evaluation: An evaluation designed to present conclusions about the merit of a program and recommendations about its future.
Formative Evaluation: An evaluation designed and used to improve a program, especially when it is still being developed.