Towards the Development of a Games-Based Learning Evaluation Framework

Towards the Development of a Games-Based Learning Evaluation Framework

Thomas Connolly (University of the West of Scotland, Scotland), Mark Stansfield (University of the West of Scotland, Scotland) and Thomas Hainey (University of the West of Scotland, Scotland)
DOI: 10.4018/978-1-60566-360-9.ch015
OnDemand PDF Download:


The field of games-based learning (GBL) has a dearth of empirical evidence supporting the validity of the approach (Connolly, Stansfield, & Hainey, 2007a; de Freitas, 2006). One primary reason for this is a distinct lack of frameworks for GBL evaluation. The literature has a wealth of articles suggesting ways that GBL can be evaluated against particular criteria with various experimental designs and analytical techniques. Based on a review of existing frameworks applicable to GBL and an extensive literature search to identify measurements that have been taken in relevant studies, this chapter will provide general guidelines to focus researchers on particular categories of evaluation, individual measurements, experimental designs and texts in the literature that have some form of empirical evidence or framework relevant to researchers evaluating GBL environments particularly focusing on learner performance. A new evaluation framework will be presented based on the compilation of all the particular areas and analytical measurements found in the literature.
Chapter Preview


One of the primary concerns associated with the GBL literature is the dearth of empirical evidence supporting the validity of the approach (Connolly, Stansfield, & Hainey, 2007a; de Freitas, 2006). O’Neil et al (2005) believe that an essential element missing is the ability to properly evaluate games for education and training purposes. If games are not properly evaluated and concrete empirical evidence is not obtained in individual learning scenarios that can produce generalisable results, then the potential of games in learning can always be dismissed as unsubstantiated optimism. In the O’Neil study, a large amount of literature was collected and analysed from the PsycINFO, EducationAbs, and SocialAciAbs information systems. Out of several thousand articles, 19 met the specified criteria for inclusion and had some kind of empirical information that was either qualitative, quantitative or both. The literature was then viewed through Kirkpatrick’s four levels for evaluating training and the augmented CRESST model. The majority of the studies reviewed analysed performance on game measurements. Other studies included observation of military tactics used, observation, time to complete the game, transfer test of position location, flight performance and a variety of questionnaires including exit, stress and motivation questionnaires.

The review of empirical evidence on the benefit of games and simulations for educational purposes is a recurring theme in the literature and can be traced even further back. For example, Randel, Morris, Wetzel, and Whitehill (1992) examined 68 studies from 1963 comparing simulations/games approaches and conventional instruction in direct relation to student performance. Some of the following main discoveries were made:

  • 38 (56%) of the studies found no difference; 22 (32%) of the studies found a difference that favoured simulations/games; 5 (7%) of studies favoured simulations/games however control was questionable; 3 (5%) found differences that favoured conventional instruction.

  • With regards to retention simulations/games induced greater retention over time than conventional techniques.

  • With regards to interest, out of 14 (21%) studies, 12 (86%) showed a greater interest in games and simulations over conventional approaches.

Although lack of empirical evidence supporting GBL is not a new issue, the growing popularity of computer games in conjunction with recent advances in games and hardware technology, the emergence of virtual worlds and massively multiplayer online games (MMOGs), reinforces the need for a flexible evaluation framework that can be used by Evaluation researchers. This chapter presents such an evaluation framework.

In the next section, we examine previous research and, in particular, discuss the types of evaluation that can be used and their applicability and importance in the field of GBL. We also examine previous evaluation frameworks that have been presented in the literature that could be applicable to GBL and follow that with the results of an extensive literature review to identify studies that performed some form of evaluation and attempted to take appropriate measurements through various experimental designs particularly focusing on learner performance. On the basis of these reviews, we then present a flexible framework for GBL evaluation from a pedagogical perspective. In the final section we discuss future validation of the new framework.

Complete Chapter List

Search this Book:
Table of Contents
Kurt Squire
Thomas Connolly, Mark Stansfield, Liz Boyle
Chapter 1
Stephen Tang, Martin Hanneghan, Abdennour El Rhalibi
Games-based learning takes advantage of gaming technologies to create a fun, motivating, and interactive virtual learning environment that promotes... Sample PDF
Introduction to Games-Based Learning
Chapter 2
Nicola Whitton
This chapter examines the rationale for the use of computer games in learning, teaching, and assessment in Higher Education. It considers their... Sample PDF
Learning and Teaching with Computer Games in Higher Education
Chapter 3
Daniel Livingstone, Jeremy Kemp, Edmund Edgar, Chris Surridge, Peter Bloomfield
Alongside the growth of interest in Games-Based Learning, there has been a notable explosion of interest in the use of 3D graphical multi-user... Sample PDF
Multi-User Virtual Environments for Learning Meet Learning Management
Chapter 4
Jean-Charles Marty, Thibault Carron, Jean-Mathias Heraud
In this chapter, the authors propose a Game-Based LMS called the pedagogical dungeon equipped with cooperation abilities for particular activities.... Sample PDF
Observation as a Requisite for Game-Based Learning Environments
Chapter 5
Marco A. Gómez-Martín, Pedro P. Gómez-Martín, Pedro A. González-Calero
A key challenge to move forward the state of the art in games-based learning systems is to facilitate instructional content creation by the domain... Sample PDF
Content Integration in Games-Based Learning Systems
Chapter 6
Matt Seeney, Helen Routledge
One of the most important differentiators between Commercial Games and Serious Games is content; delivered in a way that is successfully integrated... Sample PDF
Drawing Circles in the Sand: Integrating Content into Serious Games
Chapter 7
Mark McMahon
This chapter proposes a document-oriented instructional design model to inform the development of serious games. The model has key features in that... Sample PDF
The DODDEL Model: A Flexible Document-Oriented Model for the Design of Serious Games
Chapter 8
Daniel Burgos, Christof van Nimwegen
Serious games are suitable for learning. They are a good environment for improving the learning experience. As a key part of this setting, feedback... Sample PDF
Games-Based Learning, Destination Feedback and Adaptation: A Case Study of an Educational Planning Simulation
Chapter 9
Patrick Felicia, Ian Pitt
For a long time, users’ emotions and behaviours have been considered to obstruct rather than to help the cognitive process. Educational systems have... Sample PDF
Profiling Users in Educational Games
Chapter 10
Marco Greco
The use of Role-Playing is becoming prominent in Serious Games due to its positive effects on learning. In this chapter the author will provide a... Sample PDF
The Use of Role–Playing in Learning
Chapter 11
Sanna-Mari Tikka, Marja Kankaanranta, Tuula Nousiainen, Mari Hankala
In the context of computer games, learning is an inherent feature of computer game playing. Computer games can be seen as multimodal texts that... Sample PDF
Telling Stories with Digital Board Games: Narrative Game Worlds in Literacies Learning
Chapter 12
Colin Price
The power of computer game technology is currently being harnessed to produce “serious games”. These “games” are targeted at the education and... Sample PDF
The Path between Pedagogy and Technology: Establishing a Theoretical Basis for the Development of Educational Game Environments
Chapter 13
Sara de Freitas, Steve Jarvis
This chapter reviews some of the key research supporting the use of serious games for training in work contexts. The review indicates why serious... Sample PDF
Towards a Development Approach to Serious Games
Chapter 14
Pieter Wouters, Erik D. van der Spek, Herre van Oostendorp
Despite scant empirical substantiation, serious games are in widespread use. The authors review 28 studies with empirical data from a learning... Sample PDF
Current Practices in Serious Game Research: A Review from a Learning Outcomes Perspective
Chapter 15
Thomas Connolly, Mark Stansfield, Thomas Hainey
The field of games-based learning (GBL) has a dearth of empirical evidence supporting the validity of the approach (Connolly, Stansfield, & Hainey... Sample PDF
Towards the Development of a Games-Based Learning Evaluation Framework
Chapter 16
Helen Routledge
Based on real-world experiences using a variety of digital games, this chapter presents a guide for teachers on how to use games-based learning in... Sample PDF
Games-Based Learning in the Classroom and How it can Work!
Chapter 17
Elizabeth A. Boyle, Thomas Connolly
Developing educational computer games that will appeal to both males and females adds an additional level of complexity to an already complicated... Sample PDF
Games for Learning: Does Gender Make a Difference?
Chapter 18
Maria Saridaki, Dimitris Gouscos, Michael G. Meimaris
Students with Intellectual Disability (ID) are often described as “slow learners” and cannot easily integrate to the normal curriculum. Still, the... Sample PDF
Digital Games-Based Learning for Students with Intellectual Disability
About the Contributors