Multimedia systems waltzed into the lives of students and educators without allowing anyone the time required for the development of suitable evaluation techniques. Although everyone in the field is aware that judging this type of teaching software can only come through evaluations, the work done in this regard is scarce and ill-organized. Unfortunately, in many of the cases the evaluation forms were just filled in by instructors who pretended to be students when they went through the tutorial systems (Reiser & Kegelmann, 1994). Nowadays, however, awareness of the impact of evaluation results on the credibility of the claims made is rising.
Ever since the early days, researchers recognized the existence of two main dimensions of multimedia evaluations. Formative evaluation is concerned with the program’s functional abilities and efficiency, while summative evaluations are concerned with the effectiveness of the system in achieving its goals (Bloom, Hastings, & Madaus, 1971; Scriven, 1967).
Heller and Martin (1999) inform us that the evaluation question depends on the core subject from which it emerges, and they present a list of four subjects, namely, computer science, computer graphics, education, and human-computer interaction. They explain that if the core subject is computer science, then the research question concerns the technical requirements of the multimedia systems, including data compression, storage requirements, bandwidth, and data transmission. If the question is from computer graphics, then the focus is on speed of image rendering, representation of light, and creation of animation. If the question is from education, then media is evaluated in terms of its impact on teaching and learning along with attributes such as motivation, feedback, and information delivery. If the question is from human-computer interaction, then it is concerned with the use of multimedia in interface design focusing on issues that impact the interaction itself, such as screen design, the use of metaphor, and navigational strategies.
In retrospect, when examining their classification, we find that the main two dimensions are well covered. Formative evaluation is the focus of computer science and computer graphics as a whole, while summative evaluation is the main, but not only, focus of education and human-computer interaction. With education, we find that “motivation” does not conform to any of the two dimensions, while human-computer interaction requires both formative evaluation in addition to summative evaluation.
Researchers tested their systems through a summative evaluation frequently using a pretest and posttest where the first is taken prior to using the system and the second following the use of the system. Unfortunately, this type of testing has been plagued with no significant1 differences in student grades when multimedia is compared to classroom lectures or to carefully organized, well-illustrated textbooks (Pane, Corbett, & John, 1996). Others widened the scope of their evaluation procedure by adding learning style questionnaires that targeted student-learning preferences and a subjective questionnaire that investigated motivation issues (Kinshuk, Patel, & Russell, 2000).
Disappointment in the results of pretests and posttests caused researchers to alter the main summation evaluation question. They wondered if the test is for the educational effects of interactivity versus lack of interactivity, or should one compare animation with textual media (McKenna, 1995). If Pane et al. (1996) were aware of the work done by Freyd (1987) who studied the cognitive effects of exposing subjects to a series of still images to find that they are equivalent in the reactions they elicit to being exposed to a moving picture, then perhaps they would not have asked whether animation is equivalent to a textbook with carefully set images of all stages.
Key Terms in this Chapter
Declarative vs. Procedural Knowledge: The verbalized form of knowledge versus the implemented form of knowledge.
Cognitive Science: The field of science concerned with cognition and includes parts of cognitive psychology, linguistics, computer science cognitive neuroscience, and philosophy of mind.
Multimedia System: Any computer delivered electronic system that presents information through different media that may include text, sound, video computer graphics, and animation.
Cognitive Tool: A tool that reduces the cognitive load required by a specific task.
Cognition: The psychological result of perception, learning, and reasoning.
Cognitive Load: The degree of cognitive processes required to accomplish a specific task.
Learning Style: The manner in which an individual acquires information.