Using Expert Reviews to Enhance Learning Designs

Clarifying The Focus Of Expert Reviews In E-Learning Evaluation

This chapter rests on several well-known evaluation principles which fit together coherently:

• •

  Evaluation of e-learning is best conducted with a naturalistic approach (Guba & Lincoln, 1981). It is difficult, if not impossible, to track the actual learning outcomes of new strategies under controlled evaluation designs because of the complicated and contextual nature of educational settings. For example, it is unethical to split the class into two groups and provide different treatment to the two groups of students. As educational settings are highly multivariate, it is really impossible to control all the factors. Other evaluation strategies are needed. The expert reviews described in this chapter provide a strategy whereby informed views can be obtained on a complex artifact—a course Web site.

• •

  Authenticity, that is, evaluation in real teaching and learning contexts, is important (Oliver, 2000). Controlled experiments are often criticised as not being representative of actual classroom situations, and conclusions made from such studies are “problematic” in “generalisability” (Kember, 2003, p. 97). Our expert reviews are of ‘working’ course Web sites and not of isolated pieces of courseware.

• •

  Triangulation is essential in complex, authentic environments, and multiple sources of data are needed (Lam & McNaught, 2004). The model of evaluation that our team has developed has been used with approximately 100 educational projects in the past five years. We use data from teachers, students, and third-party reviewers in order to make judgments about educational efficiency and effectiveness. Our expert reviews are just one of a number of evaluation strategies used in the cases described.

• •

  Both qualitative and quantitative methods should be considered (Jones, Scanlon, Tosunoglu, Ross, Butcher, Murphy, & Greenberg, 1996). It is important to avoid an over-reliance on qualitative opinion data garnered from surveys and focus groups. Quantitative data, for example, from assessment results or log data, can provide useful evaluation evidence. Our expert reviews are semiquantitative in that numbers are assigned in a matrix. As we describe, this can be a trigger to discuss other qualitative feedback and design options.

• •

  Results from multiple studies provide better explanatory power (Kember, 2003). The results of a number of small studies can provide information on overall preferences and trends. One example in Hong Kong is an examination of 58 e-learning projects that indicated that glossaries, notes and PowerPoints, assessment tasks associated with grades, and exhibition of student-generated multimedia projects are considered by teachers and students in Hong Kong to be the most beneficial aspects of e-learning (McNaught & Lam, 2005). We discuss 20 Web sites in this chapter, each of which is the focus of a small-scale evaluation study.

However, it is important not to treat evaluation as a research exercise only. Another principle that underpins this chapter is that evaluation efforts should provide feedback for improvement into teaching and learning. This pragmatic focus echoes Patton’s (1997) model that evaluation should have a ‘utilization focus,’ that all stakeholders should be included in the evaluation design. Useful feedback can be provided through reports to individual teachers and also by meta-analyses across cases (Lam & McNaught, 2008; McNaught & Lam, 2005). In the work reported in this chapter both approaches are taken. In our context, therefore, the work supports individual teachers teaching their own courses and feeds into policy decision making at an institutional level.