My App is an Experiment: Experience from User Studies in Mobile App Stores

Introduction

Mobile application stores such as Apple's App Store and Google's Android Market revolutionized the distribution of applications for mobile devices. This distribution channel lowered the gateway hurdle dramatically and opened the market for small companies and engaged hobbyists. Mobile application stores -- for the first time -- enable virtually any developer to easily reach hundred thousands of mobile users. Recently researchers discovered this opportunity and began to publish prototypes via mobile application stores.

It has been argued that the “easy access to such a potentially wide audience could radically alter the nature of many UbiComp trials” (Morrison et al., 2010). In the tradition of UbiComp research most attempts to distribute prototypes via mobile application stores focus on the evaluation of prototypes (e.g. Zhai et al., 2009; Girardello, 2010; Michahelles, 2010; Gilbertson et al., 2008). Proof-of-concept prototypes are developed and the large number of users is used to demonstrate the successfulness of the respective application. Feedback is mainly gathered to understand the nature of the respective prototype.

In the tradition of psychology and social sciences Human Factors and Human-Computer Interaction research in contrast focus on understanding the human. Commonly, controlled experiments, quasi-experiments and observations are used to derive general findings. As in psychology, prototypes are often just the apparatus to investigate a research question. The psychologist Danziger describes an apparatus as a tool for “exposing experimental subjects to controlled and precisely known forms of stimulation” and “for recording and measuring responses” (Danziger & Ballantyne, 1997). In previous work we showed that Apps distributed to thousands of users can successfully be used as an apparatus for controlled experiments (Henze & Boll, 2010; Henze et al., 2010).

In this paper we report our findings from five studies we conducted by publishing Apps in the Android Market. The paper first presents these Apps, the research questions they address, and the outcomes. In the subsequent sections we then discuss our general findings and conclusions on the participants, the quantitative and qualitative data, and ethical aspects. We conclude with aspects that should be considered when conducting experiments in mobile application stores.