Collaborative Argumentation in Learning Resource Evaluation

Collaborative Argumentation in Learning Resource Evaluation

John C. Nesbit (Simon Fraser University, Canada) and Tracey L. Leacock (Simon Fraser University, Canada)
DOI: 10.4018/978-1-59904-861-1.ch028
OnDemand PDF Download:


The Learning Object Review Instrument (LORI) is an evaluation framework designed to support collaborative critique of multimedia learning resources. In this chapter, the interactions among reviewers using LORI are framed as a form of collaborative argumentation. Research on collaborative evaluation of learning resources has found that reviewers’ quality ratings tend to converge as a result of their interactions. Also, novice instructional designers have reported that collaborative evaluation is valuable preparation for undertaking resource design projects. The authors reason that collaborative evaluation is effective as a professional development method to the degree that it sustains argumentation about the application of evidence-based design principles.
Chapter Preview

Collaborative Argumentation In Learning Resource Evaluation And Design

There are several reasons why producing high quality multimedia learning resources is challenging. Many types of media, media features, and design models are available to resource developers, yet there are few standards that can guide selecting them. Relevant research on multimedia learning has expanded, yet many developers are unaware of its full scope and value. Personnel are available who specialize in media development, instructional design, usability design, subject knowledge, and teaching, yet they are rarely coordinated so that that their expertise can be effectively brought to bear. Learners usually have opinions about the resources they use, yet their opinions are rarely heard by developers.

The challenge is seen most clearly when design decisions are informed by conflicting recommendations from different specializations. Decisions about text layout are a case in point. Psychologists and educational researchers who have studied readers using computer screens to read text with a fixed number of alphabetic characters per line have observed that more characters per line (possibly up to 100) may be optimal for rapid reading, but that as few as 40 or 50 characters per line may be optimal for reading comfort and comprehension (Dyson, 2004). Ling and van Schaik (2006, p. 403) concluded that “longer line lengths should be used when information is presented that needs to be scanned quickly…. [and] shorter line lengths should be used when text is to be read more thoroughly, rather than skimmed.” Specialists familiar with this research who are designing the text components of a resource to be used for a defined learning activity might choose a fixed line length of, say, 70 characters. On the other hand, many Web developers advocate a “liquid design” for Web pages in which the number of characters per line varies according to the width of the browser window, character size, and presence of images (Weiss, 2006). They argue that readers can resize the browser window to the optimal width for normal reading, or to a much wider width that minimizes scrolling when scanning through a large document. Because neither fixed nor liquid approaches to line length is likely to be the best choice in all design situations, an analysis of how specific circumstances play into the decision seems necessary, and that process requires knowledge of both the fixed length and flexible length strategies. Finding the best design solutions and evaluating existing designs requires an exchange of specialist knowledge in relation to situated learner needs. The nature and requirements of this exchange are the concern of the present chapter.

Any approach to ensuring quality in learning objects that is built around rigid standards for technologies or implementation will quickly become obsolete. Instead, what is needed is a system for evaluating learning objects that applies design principles, recognizes that the best way to operationalize these principles will change from context to context, and has a mechanism for continued interpretation and clarification of how these principles relate to specific learning objects. We maintain that continued interpretation of quality standards requires reasoned discussion or argumentation among learning object stakeholders—media developers, instructional designers, instructors, students, and so on—and that this argumentation can also serve as a form of professional development for the stakeholders. Such dialogue provides the opportunity for professionals and students to test their ideas and see the views of other stakeholders who may be approaching the same object from different professional perspectives.

The purpose of this chapter is to present theory and evidence that collaborative argumentation can be a powerful method for the design and evaluation of multimedia learning resources. We describe how a model of collaborative argumentation that we have developed, convergent participation, has been used to evaluate learning resources and provide professional development for learning resource designers. Before taking up this main theme we introduce an instrument for evaluating multimedia learning resources that offers substantive guidance to collaborating reviewers.

Key Terms in this Chapter

Learning by Evaluation: A process in which students learn design principles by critiquing existing objects. In the course of forming and explaining their evaluation, students gain a deeper understanding of design principles than they would by only reading about them. Learning by evaluation complements learning by design in which students must create their own objects and may often be distracted by technical matters.

eLera (E-Learning Research and Assessment Network): A Web site featuring Web-based tools for evaluating learning resource quality. Members can register the metadata for any learning object and then use evaluation tools within eLera to rate the object individually or collaboratively. The goals of eLera are (1) to improve the quality of online learning resources through better design and evaluation; (2) to develop effective pedagogical models that incorporate learning objects; and (3) to help students, teachers, professors, instructional designers, and others to select pedagogical models and digital resources that meet their requirements.

Collaborative Argumentation: A form of productive critical thinking characterized by evaluation of claims and supporting evidence, consideration of alternatives, weighing of cost and benefits, and exploration of implications.

Convergent Participation: An evaluation protocol in which individuals first rate learning objects independently and then discuss the reasons for their ratings in a structured, moderated discussion. Participants may choose to change their ratings during the group discussion.

Learning Object Review Instrument (LORI): A nine-item heuristic quality rating tool for digital learning resources developed by the E-Learning Research and Assessment Network (Available from: The nine items are: content quality, learning goal alignment, feedback and adaptation, motivation, presentation design, interaction usability, accessibility, reusability, and standards compliance.

Learning Objects: Digital multimedia learning resources that combine text, images, and other media, are intended for re-use across educational settings, typically require a few minutes to perhaps an hour of a learner’s time for initial study, and usually focus on one topic or a small set of closely related elements, which could then be integrated with other objects and activities in a particular teaching context to form a full course

Complete Chapter List

Search this Book:
Table of Contents
Tom Carey
Lori Lockyer, Sue Bennett, Shirley Agostinho, Barry Harper
Lori Lockyer, Sue Bennett, Shirley Agostinho, Barry Harper
Chapter 1
Shirley Agostinho
The term “learning design” is gaining momentum in the e-learning literature as a concept for supporting academics to model and share teaching... Sample PDF
Learning Design Representations to Document, Model, and Share Teaching Practice
Chapter 2
Isobel Falconer, Allison Littlejohn
Practice models are generic approaches to the structuring and orchestration of learning activities for pedagogic purposes, intended to promote... Sample PDF
Representing Models of Practice
Chapter 3
Rob Koper, Yongwu Miao
IMS learning design (IMSLD) is an open standard that can be used to specify a wide range of pedagogical strategies in computer-interpretable models.... Sample PDF
Using the IMS LD Standard to Describe Learning Designs
Chapter 4
David Griffiths, Oleg Liber
The IMS LD specification is internally complex and has been used in a number of different ways. As a result users who have a basic understanding of... Sample PDF
Opportunities, Achievements, and Prospects for Use of IMS LD
Chapter 5
Franca Garzotto, Symeon Retalis
“A design pattern describes a problem which occurs over and over again in our environment, and then describes the core of the solution to that... Sample PDF
A Critical Perspective on Design Patterns for E-Learning
Chapter 6
Sherri S. Frizell, Roland Hübscher
Design patterns have received considerable attention for their potential as a means of capturing and sharing design knowledge. This chapter provides... Sample PDF
Using Design Patterns to Support E-Learning Design
Chapter 7
Peter Goodyear, Dai Fei Yang
This chapter provides an overview of recent research and development (R&D) activity in the area of educational design patterns and pattern... Sample PDF
Patterns and Pattern Languages in Educational Design
Chapter 8
Gráinne Conole
The chapter provides a theoretical framework for understanding learning activities, centering on two key aspects: (1) the capture and representation... Sample PDF
The Role of Mediating Artefacts in Learning Design
Chapter 9
Elizabeth Masterman
This chapter uses activity theory to construct a framework for the design and deployment of pedagogic planning tools. It starts by noting the impact... Sample PDF
Activity Theory and the Design of Pedagogic Planning Tools
Chapter 10
Barry Harper, Ron Oliver
This chapter describes the development of a taxonomy of learning designs based on a survey of 52 innovative ICT-using projects that formed the basis... Sample PDF
Developing a Taxonomy for Learning Designs
Chapter 11
Carmel McNaught, Paul Lam, Kin-Fai Cheng
The chapter will describe an expert review process used at The Chinese University of Hong Kong. The mechanism used involves a carefully developed... Sample PDF
Using Expert Reviews to Enhance Learning Designs
Chapter 12
Matthew Kearney, Anne Prescott, Kirsty Young
This chapter reports on findings from a recent project situated in the area of preservice teacher education. The project investigated prospective... Sample PDF
Investigating Prospective Teachers as Learning Design Authors
Chapter 13
Paul Hazlewood, Amanda Oddie, Mark Barrett-Baxendale
IMS Learning Design (IMS LD) is a specification for describing a range of pedagogic approaches. It allows the linking of pedagogical structure... Sample PDF
Using IMS Learning Design in Educational Situations
Chapter 14
Robert McLaughlan, Denise Kirkpatrick
Decision-making processes in relation to complex natural resources require recognition and accommodation of diverse and competing perspectives in a... Sample PDF
Online Role-Based Learning Designs for Teaching Complex Decision Making
Chapter 15
Garry Hoban
Digital animations are complex to create and are usually made by experts for novices to download from Web sites or copy from DVDs and CDs to use as... Sample PDF
Facilitating Learner-Generated Animations with Slowmation
Chapter 16
Yongwu Miao, Daniel Burgos, David Griffiths, Rob Koper
Group interaction has to be meticulously designed to foster effective and efficient collaborative learning. The IMS Learning Design specification... Sample PDF
Representation of Coordination Mechanisms in IMS LD
Chapter 17
Johannes Strobel, Gretchen Lowerison, Roger Côté, Philip C. Abrami, Edward C. Bethel
In this chapter, we describe the process of modeling different theory-, research-, and best-practicebased learning designs into IMS-LD, a... Sample PDF
Modeling Learning Units by Capturing Context with IMS LD
Chapter 18
Daniel Burgos, Hans G.K. Hummel, Colin Tattersall, Francis Brouns, Rob Koper
This chapter presents some design guidelines for collaboration and participation in blended learning networks. As an exemplary network, we describe... Sample PDF
Design Guidelines for Collaboration and Participation with Examples from the LN4LD (Learning Network for Learning Design)
Chapter 19
Tom Boyle
This chapter argues that good design has to be at the heart of developing effective learning objects. It briefly outlines the “knowledge... Sample PDF
The Design of Learning Objects for Pedagogical Impact
Chapter 20
Margaret Turner
This chapter introduces an approach to writing content for online learning over networked media. It argues that few resources currently utilise the... Sample PDF
Visual Meaning Management for Networked Learning
Chapter 21
Christina Gitsaki
Due to the increasingly diverse student population in multicultural nations such as Australia, the U.S., Canada, and the UK, educators are faced... Sample PDF
Modification of Learning Objects for NESB Students
Chapter 22
Daniel Churchill, John Gordon Hedberg
The main idea behind learning objects is that they are to exist as digital resources separated from the learning task in which they are used. This... Sample PDF
Learning Objects, Learning Tasks, and Handhelds
Chapter 23
Peter Freebody, Sandy Muspratt, David McRae
The question addressed in this chapter is: What is the evidence for the effects of online programs of learning objects on motivation and learning?... Sample PDF
Technology, Curriculum, and Pedagogy in the Evaluation of an Online Content Program in Australasia
Chapter 24
David Lake, Kate Lowe, Rob Phillips, Rick Cummings, Renato Schibeci
This chapter provides a model to analyse the effectiveness and efficiency of Learning Objects being used in primary and secondary schools by... Sample PDF
Effective Use of Learning Objects in Class Environments
Chapter 25
Robert McCormick, Tomi Jaakkola, Sami Nurmi
Most studies on reusable digital learning materials, Learning Objects (LOs), relate to their use in universities. Few empirical studies exist to... Sample PDF
A European Evaluation of the Promises of LOs
Chapter 26
Tomi Jaakkola, Sami Nurmi
There has been a clear lack of rigorous empirical evidence on the effectiveness of learning objects (LOs) in education. This chapter reports the... Sample PDF
Instructional Effectiveness of Learning Objects
Chapter 27
Robert McCormick
This chapter will examine the approach taken in the evaluation of a large-scale feasibility trial of the production, distribution, and use of... Sample PDF
Evaluating Large-Scale European LO Production, Distribution, and Use
Chapter 28
John C Nesbit, Tracey L. Leacock
The Learning Object Review Instrument (LORI) is an evaluation framework designed to support collaborative critique of multimedia learning resources.... Sample PDF
Collaborative Argumentation in Learning Resource Evaluation
Chapter 29
Philippe Martin, Michel Eboueya
This chapter first argues that current approaches for sharing and retrieving learning objects or any other kinds of information are not efficient or... Sample PDF
For the Ultimate Accessibility and Reusability
Chapter 30
Sue Bennett, Dominique Parrish, Geraldine Lefoe, Meg O’Reilly, Mike Keppell, Robyn Philip
As the notion of learning objects has grown in popularity, so too has interest in how they should be stored to promote access and reusability. A key... Sample PDF
A Needs Analysis Framework for the Design of Digital Repositories in Higher Education
Chapter 31
William Bramble, Mariya Pachman
Reusable learning objects (LOs) constitute a promising approach to the development of easily accessible, technologically sound, and curriculum... Sample PDF
Costs and Sustainability of Learning Object Repositories
Chapter 32
Kristine Elliott, Kevin Sweeney, Helen Irving
This chapter reports the authors’ experiences of developing a learning design to teach scientific inquiry, of integrating the learning design with... Sample PDF
A Learning Design to Teach Scientific Inquiry
Chapter 33
Lisa Lobry de Bruyn
This chapter explores through a case study approach of a tertiary-level unit on Land Assessment for Sustainable Use, the connections between three... Sample PDF
Adapting Problem-Based Learning to an Online Learning Environment
Chapter 34
Tan Wee Chuen, Baharuddin Aris, Mohd Salleh Abu
This chapter aims to guide the readers through the design and development of a prototype Web-based learning system based on the integration of... Sample PDF
Learning Objects and Generative Learning for Higher Order Thinking
Chapter 35
Sebastian Foti
The author describes the work of Dr. Mary Budd Rowe and the establishment of an early learning object databases. Extensive training with K-12... Sample PDF
Applying Learning Object Libraries in K-12 Settings
Chapter 36
L. K. Curda, Melissa A. Kelly
We present guidelines for designing and developing a repository for the storage and exchange of instructional resources, as well as considerations... Sample PDF
Guidelines for Developing Learning Object Repositories
Chapter 37
Sandra Wills, Anne McDougall
This study tracks the uptake of online role play in Australia from 1990 to 2006 and the affordances to its uptake. It examines reusability, as one... Sample PDF
Reusability of Online Role Play as Learning Objects or Learning Designs
Chapter 38
Lori Lockyer, Lisa Kosta, Sue Bennett
Health professional education is changing to meet the demands of a limited workforce and a focus on community-based clinical training. The change... Sample PDF
An Analysis of Learning Designs that Integrate Patient Cases in Health Professions Education
Chapter 39
Mohan Chinnappan
The shift in the way we visualise the nature of mathematics and mathematics learning has presented educational technologists with new challenges in... Sample PDF
Reconceptualisation of Learning Objects as Meta-Schemas
Chapter 40
Henk Huijser
This chapter provides an in depth discussion of the issues involved in integrating learning design and learning objects into generic Web sites. It... Sample PDF
Designing Learning Objects for Generic Web Sites
Chapter 41
Morag Munro, Claire Kenny
E-learning standards are a contentious topic amongst educators, designers, and researchers engaged in the development of learning objects and... Sample PDF
Standards for Learning Objects and Learning Designs
Chapter 42
Eddy Boot, Luca Botturi, Andrew S. Gibbons, Todd Stubbs
In developing modern instructional software, learning designs are used to formalize descriptions of roles, activities, constraints, and several... Sample PDF
Supporting Decision Making in Using Design Languages for Learning Designs and Learning Objects
Chapter 43
Gilbert Paquette, Olga Mariño, Karin Lundgren-Cayrol, Michel Léonard
This chapter summarizes the work on instructional engineering and educational modeling accomplished since 1992 at the LICEF Research Center of... Sample PDF
Principled Construction and Reuse of Learning Designs
About the Contributors