Appropriating Heuristic Evaluation for Mobile Computing

Introduction

Expert-based evaluation techniques, such as heuristic inspection (Nielsen et al., 1990) and cognitive walkthrough (Wharton et al., 1994) typically benefit from providing evaluators with guidance (for instance, a set of heuristics or a checklist) for identifying a prioritized list of usability flaws (Kjeldskov et al., 2005). Expert-based evaluation techniques are also well-known methods that can realize a relatively quick and easy evaluation.

According to Po (2003), mobile computing devices are typically ‘smart products’ or ‘information appliances’, and are generally consumer products. Their users are thus a ‘heterogeneous group’ (Sade et al., 2002) and so it may be more difficult to find suitable surrogate users for user-based testing in mobile computing (Po, 2003). Po further observes that even if appropriate surrogate users were found, the realistic re-creation of the user context in laboratories would be challenging because of user mobility, which makes observation and video recording difficult (Vetere et al., 2003). “Given the problems associated with user-based evaluations of mobile devices, expert-based usability techniques are considered to be more appropriate” (Po, 2003). However, it is worth noting that expert-based techniques have in the past been criticized for finding proportionately fewer problems in total, and disproportionately more cosmetic problems (Karat et al., 1992). In mobile computing the capacity of expert-based techniques to adequately capture the multiple contextual factors that affect user–system interactions in real settings has been questioned (for instance: Kjeldskov et al., 2003; Johnson, 1998).

We believe that heuristic evaluation can be enriched and adapted toward capturing contextual factors. This article describes how standard heuristic evaluation can be made more appropriate/relevant for mobile computing. In particular, the article describes a modified collection of usability heuristics that are designed to be appropriate for this area. The heuristics have been systematically derived from extensive literature and empirically validated, and so offer a sound basis for heuristic-based evaluation of mobile computing. As well as introducing the reader to the practical use of heuristic evaluation, the article also describes potential future research in the area.

This work has been carried out in the context of MAIS¹, a project whose research goal is to provide a flexible environment to adapt the interaction, and the information and services provided, according to ever changing requirements, execution contexts, and user needs.

The rest of the article is organized as follows: Section 2 highlights some of the challenges posed by mobile devices, applications and context; Section 3 discusses the standard heuristic evaluation method; Section 4 describes the methodology we adopted to appropriate heuristic evaluation for mobile computing and the results that we obtained; Section 5 contains reflections regarding this research activity; and Section 6 concludes the article and highlights some future work.

Our focus in this article is on usability problems in mobile devices and a discussion of their sources. However, this should be set against a broader view of the fantastic world of new opportunities, advantages, and benefits that mobile devices and contexts bring. While we will not touch explicitly on this again, the article should be read in the light that the problems and limitations are ones worth tackling because of the opportunities offered by the technology.