Article Preview
TopIntroduction
The current trend towards pervasive and context sensitive applications where information and computational technology are embedded in our social and physical environments presents substantial methodological challenges for researchers, designers, or technologists, wishing to design, analyze, or evaluate, corresponding user experiences. Available research methods have been shaped in past decades to support the design and evaluation of the cognitive ergonomics of task-oriented interaction, usually contained within a contained time span. Extending such methods to study user experiences as these occur in situ, unfolding over days or weeks, capturing social interactions between several people and diverse environmental and technical contingencies, requires a substantial scaling up the data sampling in terms of frequency, duration, and the richness of records made.
The objectives of system evaluation have also changed significantly. Transcending usability, evaluations of applications and services that are mobile and often context sensitive, typically examine higher level aspects of user experiences and user needs relating to persuasion, fun, engagement, trust, etc. Contextualized methods of data collection need to support the reporting of attitudes, opinions, or appraisals, close to the moment that a particular experience occurs and in the context where events and activities unfold. Such surveying of user attitudes can occur repeatedly over time, allowing the study of behaviors and experiences over medium or long periods of time, even enabling researchers to examine patterns of use over time.
A well established method that addresses these requirements to a large extent is the diary method whereby informants are asked to keep a journal or a log, where they record events, activities and experiences regularly over a specified period of time (Rieman, 1993). In traditional diary studies informants record data, usually in writing, but often combining or even replacing written records with other recording media, see for example Carter and Mankoff (2005).
In diary studies, the initiative for capturing information is left up to the study participants who have to remember and take the initiative to report in their diaries. This may be detrimental to the quality of the data collected for several reasons. Participants may forget to enter information in diaries, or entries may be made at moments that they have the time and appetite to do so, rather than the ones of interest to the researcher. This can lead to loss of data and systematic response biases.
For these reasons, the Experience Sampling Method (ESM) is gaining ground in human-computer interaction studies for understanding human behavior to design better products and services and for studying use in the field. The ESM is a quasi-naturalistic method that involves signaling questions at informants repeatedly throughout the sampling period. In its original form (Cziekszentimihalyi & Larson, 1987) Experience Sampling method required participants to carry a pager or any another notification device through which they would be reminded to fill in a set of questions in a paper diary . With developments in handheld computing, this method has become computerized and a variety of tools have been developed for handheld computing devices to support it (Barret & Barret, 2001). Participants are typically requested to carry a dedicated handheld device for the whole study period through which a predefined question-asking protocol is applied.
The ESM method is gaining in popularity in the field of human-computer interaction. Consolvo and Walker (2003) have used the ESM for evaluating an Intel Research system called Personal Server. Hudson et al. (2002) have used the ESM to explore attitudes about availability of managers at IBM Research. Froehlich et al. (2006) used it to investigate the relationship between explicit place ratings and implicit aspects of travel such as visit frequency. The list is longer and growing rapidly as this field turns its attention towards the design and evaluation of mobile and context sensitive applications.