The chapter will describe an expert review process used at The Chinese University of Hong Kong. The mechanism used involves a carefully developed evaluation matrix which is used with individual teachers. This matrix records: (1) the Web functions and their use as e-learning strategies in the course Web site; (2) how completely these functions are utilized; and (3) the learning design implied by the way the functions selected are used by the course documentation and gauged from conversations with the teacher. A study of 20 course Web sites in the academic years 2005–06 and 2006–07 shows that the mechanism is practical, beneficial to individual teachers, and provides data of relevance to institutional planning for e-learning.
Clarifying The Focus Of Expert Reviews In E-Learning Evaluation
This chapter rests on several well-known evaluation principles which fit together coherently:
Evaluation of e-learning is best conducted with a naturalistic approach (Guba & Lincoln, 1981). It is difficult, if not impossible, to track the actual learning outcomes of new strategies under controlled evaluation designs because of the complicated and contextual nature of educational settings. For example, it is unethical to split the class into two groups and provide different treatment to the two groups of students. As educational settings are highly multivariate, it is really impossible to control all the factors. Other evaluation strategies are needed. The expert reviews described in this chapter provide a strategy whereby informed views can be obtained on a complex artifact—a course Web site.
Authenticity, that is, evaluation in real teaching and learning contexts, is important (Oliver, 2000). Controlled experiments are often criticised as not being representative of actual classroom situations, and conclusions made from such studies are “problematic” in “generalisability” (Kember, 2003, p. 97). Our expert reviews are of ‘working’ course Web sites and not of isolated pieces of courseware.
Triangulation is essential in complex, authentic environments, and multiple sources of data are needed (Lam & McNaught, 2004). The model of evaluation that our team has developed has been used with approximately 100 educational projects in the past five years. We use data from teachers, students, and third-party reviewers in order to make judgments about educational efficiency and effectiveness. Our expert reviews are just one of a number of evaluation strategies used in the cases described.
Both qualitative and quantitative methods should be considered (Jones, Scanlon, Tosunoglu, Ross, Butcher, Murphy, & Greenberg, 1996). It is important to avoid an over-reliance on qualitative opinion data garnered from surveys and focus groups. Quantitative data, for example, from assessment results or log data, can provide useful evaluation evidence. Our expert reviews are semiquantitative in that numbers are assigned in a matrix. As we describe, this can be a trigger to discuss other qualitative feedback and design options.
Results from multiple studies provide better explanatory power (Kember, 2003). The results of a number of small studies can provide information on overall preferences and trends. One example in Hong Kong is an examination of 58 e-learning projects that indicated that glossaries, notes and PowerPoints, assessment tasks associated with grades, and exhibition of student-generated multimedia projects are considered by teachers and students in Hong Kong to be the most beneficial aspects of e-learning (McNaught & Lam, 2005). We discuss 20 Web sites in this chapter, each of which is the focus of a small-scale evaluation study.
However, it is important not to treat evaluation as a research exercise only. Another principle that underpins this chapter is that evaluation efforts should provide feedback for improvement into teaching and learning. This pragmatic focus echoes Patton’s (1997) model that evaluation should have a ‘utilization focus,’ that all stakeholders should be included in the evaluation design. Useful feedback can be provided through reports to individual teachers and also by meta-analyses across cases (Lam & McNaught, 2008; McNaught & Lam, 2005). In the work reported in this chapter both approaches are taken. In our context, therefore, the work supports individual teachers teaching their own courses and feeds into policy decision making at an institutional level.
Key Terms in this Chapter
Rule Focus: The Web is intended to enhance the teaching and explanation of knowledge and concepts.
Learning Designs: Learning designs are an amalgamation of Web functionality, learning materials/objects and/or activities, all arranged with specific learning intentions.
Role Focus: The Web is intended to support students in playing the role of a professional in the field of study. Discussion relates to ill-defined real cases and scenarios in the field and the different strategies used in different professional roles. A strong focus on immersion in authentic real-life situations.
Interactive: Students receive quite comprehensive pre-installed feedback from the computer system. This can be adaptive to students’ input. Alternatively, students may receive feedback from their peers and/or teachers.
Management: The Web is intended to facilitate class management such as online distribution of handouts and announcement of venues, special events, and so on.
Strategy Focus: The Web is intended to support students in learning how to handle ill-defined realistic problems, cases, and scenarios in the field of study. Discussion is on appropriateness of treatment and/or alternative treatments. Here, the focus is on the development of useful learning processes.
Incident Focus: The Web is intended to display well-defined real cases and scenarios. Discussion is on the incident and understanding its context.
Non-interactive: The materials on the Web are for viewing or downloading only. The computer provides no feedback or very simple (e.g., yes/no) feedback to students’ input.