Lessons Learned from Large-Scale User Studies: Using Android Market as a Source of Data

Lessons Learned from Large-Scale User Studies: Using Android Market as a Source of Data

Denzil Ferreira (University of Oulu, Finland), Vassilis Kostakos (University of Oulu, Finland) and Anind K. Dey (Carnegie Mellon University, USA)
Copyright: © 2012 |Pages: 16
DOI: 10.4018/jmhci.2012070102
OnDemand PDF Download:
No Current Special Offers


User studies with mobile devices have typically been cumbersome, since researchers have had to recruit participants, hand out or configure devices, and offer incentives and rewards. The increasing popularity of application stores has allowed researchers to use such mechanisms to recruit participants and conduct large-scale studies in authentic settings with relatively little effort. Most researchers who use application stores do not consider the side-effects or biases that such an approach may introduce. The authors summarize prior work that has reported experiences from using application stores as a recruiting, distribution and study mechanism, and also present a case study of a 4-week long study using the Android Market to deploy an application to over 4000 users that collected data on their mobile phone charging habits. The authors synthesize their own experiences with prior reported findings to discuss the challenges, advantages, limitations and considerations of using application stores as a recruitment and distribution approach for conducting large-scale studies.
Article Preview


Recruiting a large number of participants for user studies in human-computer interaction (HCI) has been challenging (e.g., participation compensation, location and time differences). Study media such as surveys and questionnaires for data collection have taken a new form in recent years, where “in the field” has been replaced with “online”, and automated logging devices have augmented diaries, video recorders and cameras (e.g., Microsoft’s SenseCam (Microsoft Research, 2007), Nokia’s LifeBlog (Nokia, 2007)). This shift represents a new trend in research methods, whereby mobile devices are used to collect data on participants and their behaviours. Distribution channels such as Google’s Android Market or Apple’s AppStore for iOS devices were established to allow users to find and install new applications easily on their devices, and now offer opportunities for researchers to deploy their own applications to facilitate their research. The popularity of mobile devices, coupled with the convenience of application stores, makes this a rather compelling and powerful mechanism for recruiting and running large-scale mobile computing studies.

Mobile devices are increasingly popular and diverse, with worldwide sales approaching 1.6 billion units, just last year (Gartner Research, 2010, 2011). Thanks to the rapid development of wireless technologies, smartphones allow people to be reachable anywhere and anytime. As “convergent” devices, smartphones empower their owners with Internet access, music, audio and video playback and recording, navigation and other communication capabilities (phone calls, SMS, MMS, etc.) (Zheng & Ni, 2006). In addition to the benefits for end users, researchers and developers can also benefit from the powerful devices that participants potentially carry on a daily basis. In the past, applications were developed by researchers on demand and deployed to a small set of participants, usually on devices provided by the researchers. Such a research method can result in misleading conclusions due to selection limitations (Oliver, 2010); not allowing users to use their own devices increases the bias that can be introduced by owning new hardware (McMillian, 2010). Nowadays, application stores allow the deployment of applications to a much wider audience, potentially on a global scale, consisting of real users who carry and own their own smart devices. As a result, researchers now can explore the potential of conducting large-scale studies without much investment in hardware or recruitment. But resorting to application stores as a distribution and recruiting mechanism has limitations and challenges of their own and is no “silver bullet” for running mobile studies where a large number of widely distributed participants are required.

This article includes a description of our use of an application store as a recruitment and distribution mechanism for conducting such a large-scale study. The discussion is grounded in both previous work and a case study summarizing our own experiences. The contribution of this article is an in-depth discussion of the challenges, advantages, limitations and considerations of using application stores as a distribution channel for conducting large-scale studies for mobile devices, grounding the discussion sections in the context of our study and its findings.

We start by summarizing related work on conducting large-scale research, followed by a description of our case study. The discussion section highlights our research results regarding our experiences running the study, the challenges and how we overcame them as well as a set of important issues related to conducting studies using application store deployments.

Complete Article List

Search this Journal:
Open Access Articles
Volume 13: 4 Issues (2021): Forthcoming, Available for Pre-Order
Volume 12: 4 Issues (2020): 3 Released, 1 Forthcoming
Volume 11: 4 Issues (2019)
Volume 10: 4 Issues (2018)
Volume 9: 4 Issues (2017)
Volume 8: 4 Issues (2016)
Volume 7: 4 Issues (2015)
Volume 6: 4 Issues (2014)
Volume 5: 4 Issues (2013)
Volume 4: 4 Issues (2012)
Volume 3: 4 Issues (2011)
Volume 2: 4 Issues (2010)
Volume 1: 4 Issues (2009)
View Complete Journal Contents Listing