Large-scale research has gained momentum in the context of Mobile Human-Computer Interaction (Mobile HCI), as many aspects of mobile app usage can only be evaluated in the real world. In this chapter, we present findings on the challenges of research in the large via app stores, in conjunction with selected data collection methods (logging, self-reporting) we identified and have proven as useful in our research. As a case study, we investigated the adoption of NFC technology, based on a gamification approach. We therefore describe the development of the game NFC Heroes involving two release cycles. We conclude with lessons learned and provide recommendations for conducting research in the large for mobile applications.
TopIntroduction
Mobile phones are one of the few truly ubiquitous interaction devices, given the sheer numbers of devices and users (and the fact that many users already have several personal portable devices, including smartphones and tablets). Given the steadily increasing number of embedded sensors of these devices, more and more options arise. Lane et al. (2010) provide a comprising survey on the potentials of mobile phones, given the plenitude of enclosed sensors, the ability to measure real world phenomena and context and to build applications on top. Phones are transferred in ‘app phones’, being served by application stores.
Exploiting this new situation of having app stores for scientific research results in significant differences to prior studies conducted ‘in the wild’: more users, more devices, more contexts, and more diversity. Researchers are no longer constrained in their studies by their lab setup, infrastructure, biases or limitations of user numbers or their diversity. Researchers now can pursue their goal to justify and prove their interaction design, application architecture, interaction metaphors or research methodology ‘in the large’, literally with no limits in participant numbers or locations. Henze (2012, p. 1) describes the problem of lab-internal studies: “Such common studies can have a high internal validity but often lack external validity. The findings cannot always be generalized to the behavior of real users in real contexts. In contrast, researchers recently started to use apps as an apparatus for mobile HCI research”. Participant databases cannot always compensate the lack of representative findings, and recruitment does not scale well as there are usually large costs associated.
As a case study on research in the large, the authors of this work developed two apps for Android phones: the NFC Heroes app and the VMI Mensa app. Leaving the safe grounds of lab-scale studies, the apps were released in the wild to allow studying the apps’ usage in context, with all the benefits and problems associated with this type of research. A ‘research in the large’ approach allows gaining insights and feedback, which is otherwise not accessible.
Outline
This chapter is structured as follows. After covering relevant work, the rationale, design, and release of NFC Heroes (“NFC Heroes on Google Play”, 2012) is comprehensively described. The game makes use of the platform’s NFC capabilities and gives users in-game incentives to scan and upload information about deployed NFC tags. The game is published via Google’s Play Store and features Facebook integration, bringing a research application to a consumer platform with the aim of realistically study usage and adoption in the wild. The conducted experiments serve as a case study for a ‘research in the large’ approach.
As further major contribution, findings are generalized and it is in more detailed reported on the experiences with selected challenges in the context of research in the large. The findings include the question of updating apps in the large and experiences with selected data collection techniques, which have been found promising in the course of this research.
The lessons learned during the process, in terms of deployment and usage, and insights gained from maintaining and updating app deployed using app stores, are summarized. This allows future research to benefit from these findings. By summarizing the findings and experiences, prior work is extended and initial results are confirmed.
Related work is discussed with a general focus on large-scale application deployments for user studies, with a focus on security, and with respect to the specific study scenario chosen in this work (i.e., NFC-based gaming).