A Framework for Structuring Learning Assessment in a Massively Multiplayer Online Educational Game: Experiment Centered Design

Introduction

Open-world games like massively multiplayer online role playing games (MMORPGs) encourage exploration and experimentation. In these environments, learning is situated in problem spaces that involve hypothesizing, probing, observing, reflecting, and recycling these steps (Gee, 2003). The open-world allows players the freedom to move and act freely within the game environment, instead of following predefined paths and action sequences (Blizzard Entertainment Inc., 2012). While research has documented how such games can be used to engage and inspire students to take interest in science, technology, engineering, and mathematical (STEM) subjects (Steinkuehler & Duncan, 2008), the field is beginning to explore how they can be used for assessment. The extended capabilities provided in MMORPGs allow for a new, innovative approach to assessment. Unlike traditional assessments, which rely on students providing itemized feedback, assessment through MMORPGs can be captured in-situ, during game play. In this paper, we will describe how unobtrusive learning assessment techniques coupled with machine learning algorithms can be utilized to record a student’s in-game actions and formulate a model of the student’s knowledge without interrupting the student’s game play. We introduce “Experiment Centered Assessment Design” (XCD), a framework for structuring a learning assessment feedback loop. XCD builds on the “Evidence Centered Assessment Design” (ECD) approach (Mislevy & Haertel, 2006), which uses tasks to elicit evidence about a student and his learning. XCD defines every task as an experiment in the scientific method, where an experiment maps a test of factors to observable outcomes. This XCD framework was applied to prototype quests in an educational MMORPG, The Radix Endeavor, being developed at The Education Arcade at the Massachusetts Institute of Technology. In the following sections, we provide background and context by first describing The Radix Endeavor. We then present an overview of learning assessment through Evidence Centered Design. Next, we describe the Experiment Centered Design assessment framework. Then we provide examples of Experiment Centered Design used in The Radix Endeavor quests. Finally, we conclude with further ideas to expand Experiment Centered Design.