Study Design and Data Gathering Guide for Serious Games’ Evaluation

Evaluation Methods For Sg Learning Outcomes

The evaluation of games is complex and multidimensional since it involves evaluation not just of whether there is an improvement in performance on the targeted learning outcomes, but also evaluation of the user acceptance of, engagement with, and satisfaction with the game. The introduction of a serious game into the curriculum raises similar issues to any other educational intervention, since the aim of a game is to improve performance on a specific learning outcome. Woolfson (2011) proposes a hierarchy of evidence for evaluating educational interventions:

• 1.

  Meta-analyses

• 2.

  Randomised controlled trials (RCT)

• 3.

  Quasi-experimental designs

• 4.

  Single case experimental designs – pre & post test

• 5.

  Non experimental designs – surveys, correlational, qualitative

Meta-Analyses: At the top of the hierarchy of evidence for the effectiveness of interventions are meta-analyses. Meta-analysis combines the results from previous studies to identify patterns in research findings, especially with respect to whether games are effective methods in learning. Meta-analysis requires a reasonable number of empirical studies as input to compare – in serious games we still have a way to go to produce the needed studies, hence it has not been included in this chapter.

Randomised Control Trials (RCT): The Randomised Control Trial (RCT) is considered to be the gold standard for evaluating educational interventions. In a RCT participants are randomly allocated to an experimental (game) group or a control (non-game) group and their performance on the target skill/behaviour before and after the game intervention is tested. Ideally pre-testing should confirm no existing difference between the groups, while post-testing should show whether the experimental group performs better than the control group. Improvements in the target skill/behaviour for the experimental compared with the control group in a follow-up study would allow further confirmation that the intervention was successful.