A Cognitively-Based Framework for Evaluating Multimedia Systems

Background

Ever since the early days, researchers recognized the existence of two main dimensions of multimedia evaluations. Formative evaluation is concerned with the program’s functional abilities and efficiency, while summative evaluations are concerned with the effectiveness of the system in achieving its goals (Bloom, Hastings, & Madaus, 1971; Scriven, 1967).

Heller and Martin (1999) inform us that the evaluation question depends on the core subject from which it emerges, and they present a list of four subjects, namely, computer science, computer graphics, education, and human-computer interaction. They explain that if the core subject is computer science, then the research question concerns the technical requirements of the multimedia systems, including data compression, storage requirements, bandwidth, and data transmission. If the question is from computer graphics, then the focus is on speed of image rendering, representation of light, and creation of animation. If the question is from education, then media is evaluated in terms of its impact on teaching and learning along with attributes such as motivation, feedback, and information delivery. If the question is from human-computer interaction, then it is concerned with the use of multimedia in interface design focusing on issues that impact the interaction itself, such as screen design, the use of metaphor, and navigational strategies.

In retrospect, when examining their classification, we find that the main two dimensions are well covered. Formative evaluation is the focus of computer science and computer graphics as a whole, while summative evaluation is the main, but not only, focus of education and human-computer interaction. With education, we find that “motivation” does not conform to any of the two dimensions, while human-computer interaction requires both formative evaluation in addition to summative evaluation.

Researchers tested their systems through a summative evaluation frequently using a pretest and posttest where the first is taken prior to using the system and the second following the use of the system. Unfortunately, this type of testing has been plagued with no significant¹ differences in student grades when multimedia is compared to classroom lectures or to carefully organized, well-illustrated textbooks (Pane, Corbett, & John, 1996). Others widened the scope of their evaluation procedure by adding learning style questionnaires that targeted student-learning preferences and a subjective questionnaire that investigated motivation issues (Kinshuk, Patel, & Russell, 2000).

Disappointment in the results of pretests and posttests caused researchers to alter the main summation evaluation question. They wondered if the test is for the educational effects of interactivity versus lack of interactivity, or should one compare animation with textual media (McKenna, 1995). If Pane et al. (1996) were aware of the work done by Freyd (1987) who studied the cognitive effects of exposing subjects to a series of still images to find that they are equivalent in the reactions they elicit to being exposed to a moving picture, then perhaps they would not have asked whether animation is equivalent to a textbook with carefully set images of all stages.