Meeting the Challenges in Evaluating Mobile Learning: A 3-Level Evaluation Framework

Challenge 1: Capturing Learning Context And Learning Across Contexts

A major task for educational evaluation is to identify and analyse learning within and across contexts. For mobile learning, the interest is not only in how learning occurs in a variety of settings, but also how people create new contexts for learning through their interactions and how they progress learning across contexts. This poses a significant challenge to evaluators of mobile learning. In order to establish, document and evaluate learning within and across contexts, a researcher needs to analyse: the physical setting and the layout of the learning space (where); the social setting (who, with whom, from whom); the learning objectives and outcomes (why and what); the learning methods and activities (how); the learning progress and history (when); and the learning tools (how).

When evaluating learning in a traditional classroom, researchers generally have access to information about these context elements before, during and after the learning experience. Thus, they can inspect the classroom and interview the teacher and learners in advance of a lesson to discover the objectives, methods, lesson plan and tools. To evaluate a school museum visit or field trip, the researcher can visit the site and inspect the lesson plan, but will generally not know in advance the route that each student will take. For personal or family visits to museums or other learning sites, neither the objectives nor the trajectory may be known in advance. Learning objectives may arise as a response to interactions with the environment and learning trails may be guided by curiosity or unplanned events. The learners themselves may not be known in advance, for example when evaluating the learning experience of museum visitors randomly selected at the museum entrance. Personal mobile learning embraces any learning event where people, individually and collectively, continually create micro-sites for learning out of the available physical and social resources. In considering this generic case, the setting, objectives, methods and processes may all be unpredictable.

Table 1 portrays the increasing vagueness in moving from evaluating a classroom lesson, to a school museum visit, to personal or family museum visits, to personal mobile learning across formal and informal settings. Each set of context elements requires specific evaluation methods, to match the actual learning processes and outcomes to expectations, or to capture contingent and unexpected learning events.