Using Expert Reviews to Enhance Learning Designs

Using Expert Reviews to Enhance Learning Designs

Carmel McNaught (The Chinese University of Hong Kong, Hong Kong), Paul Lam (The Chinese University of Hong Kong, Hong Kong) and Kin-Fai Cheng (The Chinese University of Hong Kong, Hong Kong)
DOI: 10.4018/978-1-59904-861-1.ch011
OnDemand PDF Download:


The chapter will describe an expert review process used at The Chinese University of Hong Kong. The mechanism used involves a carefully developed evaluation matrix which is used with individual teachers. This matrix records: (1) the Web functions and their use as e-learning strategies in the course Web site; (2) how completely these functions are utilized; and (3) the learning design implied by the way the functions selected are used by the course documentation and gauged from conversations with the teacher. A study of 20 course Web sites in the academic years 2005–06 and 2006–07 shows that the mechanism is practical, beneficial to individual teachers, and provides data of relevance to institutional planning for e-learning.
Chapter Preview

Clarifying The Focus Of Expert Reviews In E-Learning Evaluation

This chapter rests on several well-known evaluation principles which fit together coherently:

  • Evaluation of e-learning is best conducted with a naturalistic approach (Guba & Lincoln, 1981). It is difficult, if not impossible, to track the actual learning outcomes of new strategies under controlled evaluation designs because of the complicated and contextual nature of educational settings. For example, it is unethical to split the class into two groups and provide different treatment to the two groups of students. As educational settings are highly multivariate, it is really impossible to control all the factors. Other evaluation strategies are needed. The expert reviews described in this chapter provide a strategy whereby informed views can be obtained on a complex artifact—a course Web site.

  • Authenticity, that is, evaluation in real teaching and learning contexts, is important (Oliver, 2000). Controlled experiments are often criticised as not being representative of actual classroom situations, and conclusions made from such studies are “problematic” in “generalisability” (Kember, 2003, p. 97). Our expert reviews are of ‘working’ course Web sites and not of isolated pieces of courseware.

  • Triangulation is essential in complex, authentic environments, and multiple sources of data are needed (Lam & McNaught, 2004). The model of evaluation that our team has developed has been used with approximately 100 educational projects in the past five years. We use data from teachers, students, and third-party reviewers in order to make judgments about educational efficiency and effectiveness. Our expert reviews are just one of a number of evaluation strategies used in the cases described.

  • Both qualitative and quantitative methods should be considered (Jones, Scanlon, Tosunoglu, Ross, Butcher, Murphy, & Greenberg, 1996). It is important to avoid an over-reliance on qualitative opinion data garnered from surveys and focus groups. Quantitative data, for example, from assessment results or log data, can provide useful evaluation evidence. Our expert reviews are semiquantitative in that numbers are assigned in a matrix. As we describe, this can be a trigger to discuss other qualitative feedback and design options.

  • Results from multiple studies provide better explanatory power (Kember, 2003). The results of a number of small studies can provide information on overall preferences and trends. One example in Hong Kong is an examination of 58 e-learning projects that indicated that glossaries, notes and PowerPoints, assessment tasks associated with grades, and exhibition of student-generated multimedia projects are considered by teachers and students in Hong Kong to be the most beneficial aspects of e-learning (McNaught & Lam, 2005). We discuss 20 Web sites in this chapter, each of which is the focus of a small-scale evaluation study.

However, it is important not to treat evaluation as a research exercise only. Another principle that underpins this chapter is that evaluation efforts should provide feedback for improvement into teaching and learning. This pragmatic focus echoes Patton’s (1997) model that evaluation should have a ‘utilization focus,’ that all stakeholders should be included in the evaluation design. Useful feedback can be provided through reports to individual teachers and also by meta-analyses across cases (Lam & McNaught, 2008; McNaught & Lam, 2005). In the work reported in this chapter both approaches are taken. In our context, therefore, the work supports individual teachers teaching their own courses and feeds into policy decision making at an institutional level.

Key Terms in this Chapter

Rule Focus: The Web is intended to enhance the teaching and explanation of knowledge and concepts.

Learning Designs: Learning designs are an amalgamation of Web functionality, learning materials/objects and/or activities, all arranged with specific learning intentions.

Role Focus: The Web is intended to support students in playing the role of a professional in the field of study. Discussion relates to ill-defined real cases and scenarios in the field and the different strategies used in different professional roles. A strong focus on immersion in authentic real-life situations.

Interactive: Students receive quite comprehensive pre-installed feedback from the computer system. This can be adaptive to students’ input. Alternatively, students may receive feedback from their peers and/or teachers.

Management: The Web is intended to facilitate class management such as online distribution of handouts and announcement of venues, special events, and so on.

Strategy Focus: The Web is intended to support students in learning how to handle ill-defined realistic problems, cases, and scenarios in the field of study. Discussion is on appropriateness of treatment and/or alternative treatments. Here, the focus is on the development of useful learning processes.

Incident Focus: The Web is intended to display well-defined real cases and scenarios. Discussion is on the incident and understanding its context.

Non-interactive: The materials on the Web are for viewing or downloading only. The computer provides no feedback or very simple (e.g., yes/no) feedback to students’ input.

Complete Chapter List

Search this Book:
Table of Contents
Tom Carey
Lori Lockyer, Sue Bennett, Shirley Agostinho, Barry Harper
Lori Lockyer, Sue Bennett, Shirley Agostinho, Barry Harper
Chapter 1
Shirley Agostinho
The term “learning design” is gaining momentum in the e-learning literature as a concept for supporting academics to model and share teaching... Sample PDF
Learning Design Representations to Document, Model, and Share Teaching Practice
Chapter 2
Isobel Falconer, Allison Littlejohn
Practice models are generic approaches to the structuring and orchestration of learning activities for pedagogic purposes, intended to promote... Sample PDF
Representing Models of Practice
Chapter 3
Rob Koper, Yongwu Miao
IMS learning design (IMSLD) is an open standard that can be used to specify a wide range of pedagogical strategies in computer-interpretable models.... Sample PDF
Using the IMS LD Standard to Describe Learning Designs
Chapter 4
David Griffiths, Oleg Liber
The IMS LD specification is internally complex and has been used in a number of different ways. As a result users who have a basic understanding of... Sample PDF
Opportunities, Achievements, and Prospects for Use of IMS LD
Chapter 5
Franca Garzotto, Symeon Retalis
“A design pattern describes a problem which occurs over and over again in our environment, and then describes the core of the solution to that... Sample PDF
A Critical Perspective on Design Patterns for E-Learning
Chapter 6
Sherri S. Frizell, Roland Hübscher
Design patterns have received considerable attention for their potential as a means of capturing and sharing design knowledge. This chapter provides... Sample PDF
Using Design Patterns to Support E-Learning Design
Chapter 7
Peter Goodyear, Dai Fei Yang
This chapter provides an overview of recent research and development (R&D) activity in the area of educational design patterns and pattern... Sample PDF
Patterns and Pattern Languages in Educational Design
Chapter 8
Gráinne Conole
The chapter provides a theoretical framework for understanding learning activities, centering on two key aspects: (1) the capture and representation... Sample PDF
The Role of Mediating Artefacts in Learning Design
Chapter 9
Elizabeth Masterman
This chapter uses activity theory to construct a framework for the design and deployment of pedagogic planning tools. It starts by noting the impact... Sample PDF
Activity Theory and the Design of Pedagogic Planning Tools
Chapter 10
Barry Harper, Ron Oliver
This chapter describes the development of a taxonomy of learning designs based on a survey of 52 innovative ICT-using projects that formed the basis... Sample PDF
Developing a Taxonomy for Learning Designs
Chapter 11
Carmel McNaught, Paul Lam, Kin-Fai Cheng
The chapter will describe an expert review process used at The Chinese University of Hong Kong. The mechanism used involves a carefully developed... Sample PDF
Using Expert Reviews to Enhance Learning Designs
Chapter 12
Matthew Kearney, Anne Prescott, Kirsty Young
This chapter reports on findings from a recent project situated in the area of preservice teacher education. The project investigated prospective... Sample PDF
Investigating Prospective Teachers as Learning Design Authors
Chapter 13
Paul Hazlewood, Amanda Oddie, Mark Barrett-Baxendale
IMS Learning Design (IMS LD) is a specification for describing a range of pedagogic approaches. It allows the linking of pedagogical structure... Sample PDF
Using IMS Learning Design in Educational Situations
Chapter 14
Robert McLaughlan, Denise Kirkpatrick
Decision-making processes in relation to complex natural resources require recognition and accommodation of diverse and competing perspectives in a... Sample PDF
Online Role-Based Learning Designs for Teaching Complex Decision Making
Chapter 15
Garry Hoban
Digital animations are complex to create and are usually made by experts for novices to download from Web sites or copy from DVDs and CDs to use as... Sample PDF
Facilitating Learner-Generated Animations with Slowmation
Chapter 16
Yongwu Miao, Daniel Burgos, David Griffiths, Rob Koper
Group interaction has to be meticulously designed to foster effective and efficient collaborative learning. The IMS Learning Design specification... Sample PDF
Representation of Coordination Mechanisms in IMS LD
Chapter 17
Johannes Strobel, Gretchen Lowerison, Roger Côté, Philip C. Abrami, Edward C. Bethel
In this chapter, we describe the process of modeling different theory-, research-, and best-practicebased learning designs into IMS-LD, a... Sample PDF
Modeling Learning Units by Capturing Context with IMS LD
Chapter 18
Daniel Burgos, Hans G.K. Hummel, Colin Tattersall, Francis Brouns, Rob Koper
This chapter presents some design guidelines for collaboration and participation in blended learning networks. As an exemplary network, we describe... Sample PDF
Design Guidelines for Collaboration and Participation with Examples from the LN4LD (Learning Network for Learning Design)
Chapter 19
Tom Boyle
This chapter argues that good design has to be at the heart of developing effective learning objects. It briefly outlines the “knowledge... Sample PDF
The Design of Learning Objects for Pedagogical Impact
Chapter 20
Margaret Turner
This chapter introduces an approach to writing content for online learning over networked media. It argues that few resources currently utilise the... Sample PDF
Visual Meaning Management for Networked Learning
Chapter 21
Christina Gitsaki
Due to the increasingly diverse student population in multicultural nations such as Australia, the U.S., Canada, and the UK, educators are faced... Sample PDF
Modification of Learning Objects for NESB Students
Chapter 22
Daniel Churchill, John Gordon Hedberg
The main idea behind learning objects is that they are to exist as digital resources separated from the learning task in which they are used. This... Sample PDF
Learning Objects, Learning Tasks, and Handhelds
Chapter 23
Peter Freebody, Sandy Muspratt, David McRae
The question addressed in this chapter is: What is the evidence for the effects of online programs of learning objects on motivation and learning?... Sample PDF
Technology, Curriculum, and Pedagogy in the Evaluation of an Online Content Program in Australasia
Chapter 24
David Lake, Kate Lowe, Rob Phillips, Rick Cummings, Renato Schibeci
This chapter provides a model to analyse the effectiveness and efficiency of Learning Objects being used in primary and secondary schools by... Sample PDF
Effective Use of Learning Objects in Class Environments
Chapter 25
Robert McCormick, Tomi Jaakkola, Sami Nurmi
Most studies on reusable digital learning materials, Learning Objects (LOs), relate to their use in universities. Few empirical studies exist to... Sample PDF
A European Evaluation of the Promises of LOs
Chapter 26
Tomi Jaakkola, Sami Nurmi
There has been a clear lack of rigorous empirical evidence on the effectiveness of learning objects (LOs) in education. This chapter reports the... Sample PDF
Instructional Effectiveness of Learning Objects
Chapter 27
Robert McCormick
This chapter will examine the approach taken in the evaluation of a large-scale feasibility trial of the production, distribution, and use of... Sample PDF
Evaluating Large-Scale European LO Production, Distribution, and Use
Chapter 28
John C Nesbit, Tracey L. Leacock
The Learning Object Review Instrument (LORI) is an evaluation framework designed to support collaborative critique of multimedia learning resources.... Sample PDF
Collaborative Argumentation in Learning Resource Evaluation
Chapter 29
Philippe Martin, Michel Eboueya
This chapter first argues that current approaches for sharing and retrieving learning objects or any other kinds of information are not efficient or... Sample PDF
For the Ultimate Accessibility and Reusability
Chapter 30
Sue Bennett, Dominique Parrish, Geraldine Lefoe, Meg O’Reilly, Mike Keppell, Robyn Philip
As the notion of learning objects has grown in popularity, so too has interest in how they should be stored to promote access and reusability. A key... Sample PDF
A Needs Analysis Framework for the Design of Digital Repositories in Higher Education
Chapter 31
William Bramble, Mariya Pachman
Reusable learning objects (LOs) constitute a promising approach to the development of easily accessible, technologically sound, and curriculum... Sample PDF
Costs and Sustainability of Learning Object Repositories
Chapter 32
Kristine Elliott, Kevin Sweeney, Helen Irving
This chapter reports the authors’ experiences of developing a learning design to teach scientific inquiry, of integrating the learning design with... Sample PDF
A Learning Design to Teach Scientific Inquiry
Chapter 33
Lisa Lobry de Bruyn
This chapter explores through a case study approach of a tertiary-level unit on Land Assessment for Sustainable Use, the connections between three... Sample PDF
Adapting Problem-Based Learning to an Online Learning Environment
Chapter 34
Tan Wee Chuen, Baharuddin Aris, Mohd Salleh Abu
This chapter aims to guide the readers through the design and development of a prototype Web-based learning system based on the integration of... Sample PDF
Learning Objects and Generative Learning for Higher Order Thinking
Chapter 35
Sebastian Foti
The author describes the work of Dr. Mary Budd Rowe and the establishment of an early learning object databases. Extensive training with K-12... Sample PDF
Applying Learning Object Libraries in K-12 Settings
Chapter 36
L. K. Curda, Melissa A. Kelly
We present guidelines for designing and developing a repository for the storage and exchange of instructional resources, as well as considerations... Sample PDF
Guidelines for Developing Learning Object Repositories
Chapter 37
Sandra Wills, Anne McDougall
This study tracks the uptake of online role play in Australia from 1990 to 2006 and the affordances to its uptake. It examines reusability, as one... Sample PDF
Reusability of Online Role Play as Learning Objects or Learning Designs
Chapter 38
Lori Lockyer, Lisa Kosta, Sue Bennett
Health professional education is changing to meet the demands of a limited workforce and a focus on community-based clinical training. The change... Sample PDF
An Analysis of Learning Designs that Integrate Patient Cases in Health Professions Education
Chapter 39
Mohan Chinnappan
The shift in the way we visualise the nature of mathematics and mathematics learning has presented educational technologists with new challenges in... Sample PDF
Reconceptualisation of Learning Objects as Meta-Schemas
Chapter 40
Henk Huijser
This chapter provides an in depth discussion of the issues involved in integrating learning design and learning objects into generic Web sites. It... Sample PDF
Designing Learning Objects for Generic Web Sites
Chapter 41
Morag Munro, Claire Kenny
E-learning standards are a contentious topic amongst educators, designers, and researchers engaged in the development of learning objects and... Sample PDF
Standards for Learning Objects and Learning Designs
Chapter 42
Eddy Boot, Luca Botturi, Andrew S. Gibbons, Todd Stubbs
In developing modern instructional software, learning designs are used to formalize descriptions of roles, activities, constraints, and several... Sample PDF
Supporting Decision Making in Using Design Languages for Learning Designs and Learning Objects
Chapter 43
Gilbert Paquette, Olga Mariño, Karin Lundgren-Cayrol, Michel Léonard
This chapter summarizes the work on instructional engineering and educational modeling accomplished since 1992 at the LICEF Research Center of... Sample PDF
Principled Construction and Reuse of Learning Designs
About the Contributors