Experiments are a cornerstone of HCI research. Mobile distribution channels such as Apple’s App Store and Google’s Android Market have created the opportunity to bring experiments to the end user. Hardly any experience exists on how to conduct such experiments successfully. This article reports on five experiments that were conducted by publishing Apps in the Android Market. The Apps are freely available and have been installed more than 30,000 times. The outcomes of the experiments range from failure to valuable insights. Based on these outcomes, the authors identified factors that account for the success of experiments using mobile application stores. When generalizing findings it must be considered that smartphone users are a non-representative sample of the world’s population. Most participants can be obtained by informing users about the study when the App had been started for the first time. Because Apps are often used for a short time only, data should be collected as early as possible. To collect valuable qualitative feedback other channels than user comments and email have to be used. Finally, the interpretation of collected data has to consider unpredicted usage patterns to provide valid conclusions.
TopIntroduction
Mobile application stores such as Apple's App Store and Google's Android Market revolutionized the distribution of applications for mobile devices. This distribution channel lowered the gateway hurdle dramatically and opened the market for small companies and engaged hobbyists. Mobile application stores -- for the first time -- enable virtually any developer to easily reach hundred thousands of mobile users. Recently researchers discovered this opportunity and began to publish prototypes via mobile application stores.
It has been argued that the “easy access to such a potentially wide audience could radically alter the nature of many UbiComp trials” (Morrison et al., 2010). In the tradition of UbiComp research most attempts to distribute prototypes via mobile application stores focus on the evaluation of prototypes (e.g. Zhai et al., 2009; Girardello, 2010; Michahelles, 2010; Gilbertson et al., 2008). Proof-of-concept prototypes are developed and the large number of users is used to demonstrate the successfulness of the respective application. Feedback is mainly gathered to understand the nature of the respective prototype.
In the tradition of psychology and social sciences Human Factors and Human-Computer Interaction research in contrast focus on understanding the human. Commonly, controlled experiments, quasi-experiments and observations are used to derive general findings. As in psychology, prototypes are often just the apparatus to investigate a research question. The psychologist Danziger describes an apparatus as a tool for “exposing experimental subjects to controlled and precisely known forms of stimulation” and “for recording and measuring responses” (Danziger & Ballantyne, 1997). In previous work we showed that Apps distributed to thousands of users can successfully be used as an apparatus for controlled experiments (Henze & Boll, 2010; Henze et al., 2010).
In this paper we report our findings from five studies we conducted by publishing Apps in the Android Market. The paper first presents these Apps, the research questions they address, and the outcomes. In the subsequent sections we then discuss our general findings and conclusions on the participants, the quantitative and qualitative data, and ethical aspects. We conclude with aspects that should be considered when conducting experiments in mobile application stores.
Case Studies
In order to investigate different mobile HCI topics we conducted five studies, which actually use an App as apparatus and were published via mobile application stores. All Apps have been implemented for the Android platform and are therefore available for a range of users and devices. Table 1 gives an overview of the studies that are described in the following.
Table 1. Overview about the five conducted studied
Name | Installs | Samples | Time | Type |
SINLA | ~1,737 | 8 | 8.5 months | quasi-experiment |
PocketNavigator | 9,149 | 670 | 6 months | quasi-experiment |
MapExplorer | 6,372 | 4,197 | 6 months | experiment |
Poke the Rabbit | 5,708 | 5,103 | 5.5 months | experiment |
Tap It | 7,811 | 6,907 | 2 months | observation |