Experimental Setups for User Evaluation of Mobile Devices and Ubiquitous Systems

Abstract

Nowadays, mobile devices features are often linked up to the context of usage. As a consequence, researchers must consider not only the user and the device, but also the surrounding environment when designing effective user study evaluations. Two opposite experimental setups are possible: in-situ and in the laboratory. There is no consensus on their respective benefits, for instance with regard to the number of usability issues detected. In this chapter, the author isolates independent variables that could contribute to evaluation biases by proposing a taxonomy that splits the in-situ and laboratory experimental setups into two new setups. The author describes the concept of the “Uncertainty Principle” to emphasize the dilemma between precise observation and bias minimization and introduce the “Trojan Horse” technique to partially overcome the consequences of the uncertainty principle. As a conclusion, a methodology using the four experimental setups in a complementary way is proposed.