Use of Scripted Role-Play in Evaluation of Multiple-User Multiple-Service Mobile Social and Pervasive Systems

Use of Scripted Role-Play in Evaluation of Multiple-User Multiple-Service Mobile Social and Pervasive Systems

Edel Jennings (TSSG, Waterford Institute of Technology, Waterford, Ireland), Mark Roddy (TSSG, Waterford Institute of Technology, Waterford, Ireland), Alexander J. Leckey (Intel Labs Europe, Leixlip, Ireland) and Guy Feigenblat (IBM Research, Haifa, Israel)
Copyright: © 2015 |Pages: 18
DOI: 10.4018/IJMHCI.2015100103


Mobile and social computing is rapidly evolving towards a deeper integration with the physical world due to the proliferation of smart connected objects. It is widely acknowledged that involving end users in the design, development and evaluation of applications that function within the resulting complex socio-technical systems is crucial. However, reliable methods for managing evaluation of medium fidelity prototypes, whose utility is often dependent on rich data sets and/or the presence of multiple users simultaneously engaging in multiple activities, have not yet emerged. The authors report on the use of scripted role-play as an experimental approach applied in a mixed-methods evaluation of early prototypes of a suite of professional networking applications targeting a conference attendance scenario. Their evaluation was significantly constrained by the limited availability of a small cohort of end users for a relatively short period of time, which pose a challenge to define interactions that would ensure these users could experience and understand the novel application features. The authors observed that participatory role-play facilitated deeper user engagement with, exploration of, and discussion about, the mobile social applications than would have been possible with traditional usability approaches given the small user cohort and the time-constrained conditions.
Article Preview


As smart phones become almost ubiquitous, User Experience (UX) is increasingly recognised as a significant differentiator for applications in a crowded marketplace. Thus, the inclusion of User Centered Design (UCD) processes which can deliver excellent user experience is ever more significant in the design and development of mobile and social Information and Technology (ICT) services. Understanding users’ expectations, experiences and perceptions is particularly important in the case of context-aware pervasive computing, where context-of-use is directly meaningful for human system interactions.

While traditional usability approaches, such as Think Aloud (Wright & Monk, 1991), or Cognitive Walkthrough (Wharton, Rieman, Lewis, & Polson, 1994), evolved to learn about one user’s interactions with a particular pre-defined set of tasks in one computing application, these and other usability approaches are not easily adapted for multiple-user multiple-service mobile social and pervasive systems, such as the SOCIETIES platform and third party services which we use for a case study discussion in this paper. Attempting to list tasks and functionalities is more complex in the case of multiple user social and context aware systems, as network effects such as feedback and community dynamics and creative appropriation (which are properties that emerge in such systems through relevant data population and usage) are not easily modeled. Social and pervasive systems are evolutionary, and data gathered and generated through monitoring users’ actions in particular contexts, and interactions with services and other users is: 1) particular to the specific users, enabling personalisation; and 2) significant to the emergence of opportunities for sharing of data, services and things offered through the identification of shared interests across communities of users in particular contexts. Therefore, it is difficult to present random users with opportunities that feel sufficiently authentic for them to experience the system from within.

Large extended field trials are often proposed as the preferred evaluation approach for context aware pervasive or social systems using mobile applications. Field trials allow for ecological validity which is important for complex social technical systems—which means facilitating trial participants’ natural usage of services in everyday life, rather than in controlled laboratory conditions (Eagle & Pentland, 2006; Oulasvirta, 2011; Rogers et al., 2007a). It also implies participants’ interactions with the services and systems under investigation will be competing for attention with the myriad of distractions and interruptions associated with active usage ‘in the wild’. However, in the case of early prototypes, which are not yet sufficiently robust or mature for extended deployment with multiple users, more creative solutions are required.

In order to evaluate two medium fidelity Enterprise-focused services designed for a conference scenario, based on a novel social and pervasive platform, with a small number of participants in a workshop setting over the course of one afternoon, we elected to use scripted role-play, combined with other participatory demonstrations and games. A mixed methods approach, including surveys, cognitive walkthrough, games, role play and focus group discussion, afforded us the necessary flexibility and adaptability to glean insights and invite feedback from participants at various different points of the evaluation, from initial expectations and impressions, to immersion in usage, and final considered reflections. Role-play allowed us to simulate situated use and condense conference type interactions to a few hours. We found that it enabled participants to engage in-depth with the presented technologies, whilst also facilitating developers to gain useful insights.

Complete Article List

Search this Journal:
Open Access Articles
Volume 11: 4 Issues (2019): Forthcoming, Available for Pre-Order
Volume 10: 4 Issues (2018)
Volume 9: 4 Issues (2017)
Volume 8: 4 Issues (2016)
Volume 7: 4 Issues (2015)
Volume 6: 4 Issues (2014)
Volume 5: 4 Issues (2013)
Volume 4: 4 Issues (2012)
Volume 3: 4 Issues (2011)
Volume 2: 4 Issues (2010)
Volume 1: 4 Issues (2009)
View Complete Journal Contents Listing