Article Preview
TopIntroduction
Recruiting a large number of participants for user studies in human-computer interaction (HCI) has been challenging (e.g., participation compensation, location and time differences). Study media such as surveys and questionnaires for data collection have taken a new form in recent years, where “in the field” has been replaced with “online”, and automated logging devices have augmented diaries, video recorders and cameras (e.g., Microsoft’s SenseCam (Microsoft Research, 2007), Nokia’s LifeBlog (Nokia, 2007)). This shift represents a new trend in research methods, whereby mobile devices are used to collect data on participants and their behaviours. Distribution channels such as Google’s Android Market or Apple’s AppStore for iOS devices were established to allow users to find and install new applications easily on their devices, and now offer opportunities for researchers to deploy their own applications to facilitate their research. The popularity of mobile devices, coupled with the convenience of application stores, makes this a rather compelling and powerful mechanism for recruiting and running large-scale mobile computing studies.
Mobile devices are increasingly popular and diverse, with worldwide sales approaching 1.6 billion units, just last year (Gartner Research, 2010, 2011). Thanks to the rapid development of wireless technologies, smartphones allow people to be reachable anywhere and anytime. As “convergent” devices, smartphones empower their owners with Internet access, music, audio and video playback and recording, navigation and other communication capabilities (phone calls, SMS, MMS, etc.) (Zheng & Ni, 2006). In addition to the benefits for end users, researchers and developers can also benefit from the powerful devices that participants potentially carry on a daily basis. In the past, applications were developed by researchers on demand and deployed to a small set of participants, usually on devices provided by the researchers. Such a research method can result in misleading conclusions due to selection limitations (Oliver, 2010); not allowing users to use their own devices increases the bias that can be introduced by owning new hardware (McMillian, 2010). Nowadays, application stores allow the deployment of applications to a much wider audience, potentially on a global scale, consisting of real users who carry and own their own smart devices. As a result, researchers now can explore the potential of conducting large-scale studies without much investment in hardware or recruitment. But resorting to application stores as a distribution and recruiting mechanism has limitations and challenges of their own and is no “silver bullet” for running mobile studies where a large number of widely distributed participants are required.
This article includes a description of our use of an application store as a recruitment and distribution mechanism for conducting such a large-scale study. The discussion is grounded in both previous work and a case study summarizing our own experiences. The contribution of this article is an in-depth discussion of the challenges, advantages, limitations and considerations of using application stores as a distribution channel for conducting large-scale studies for mobile devices, grounding the discussion sections in the context of our study and its findings.
We start by summarizing related work on conducting large-scale research, followed by a description of our case study. The discussion section highlights our research results regarding our experiences running the study, the challenges and how we overcame them as well as a set of important issues related to conducting studies using application store deployments.