Traditional user experience assessments rely on self-report, human-system performance, and observational data that incompletely capture users' psychological demands, processing, or affect. Specifically, self-report measures require users to identify and articulate subjective responses to product features, yet users may not possess accurate awareness or may be unwilling or unable to express themselves. Similarly, human-system performance and observational measures require analysts to make inferences about hidden psychological states based on observed external patterns. This chapter discusses how biometric sensor-based affect detection technologies (e.g., eye tracking and EEG) may supplement traditional methods. By measuring biometric indicators of psychological states, researchers can gain potentially richer and more accurate insights into user experience. These technologies are gaining traction in educational technology development and functionality, and thus the extension of these tools for usability and user experience evaluation is highly feasible.
TopIntroduction
User experience broadly refers to users’ behaviors, thoughts, and feelings while interacting with product features, which occur in response to design features including aesthetics, multimedia content, navigation, and interactivity (see Baxter, Courage, & Caine, 2015; Tullis & Albert, 2013; Zhang & Adipat, 2005, for various definitions of user experience). To assess these experiences and interactions, analysts have access to a variety of methods. For instance, users’ thoughts and feelings can be probed as they conduct tasks by asking users to think-aloud (Ericsson & Simon, 1993; Van Den Haak, De Jong, & Jan Schellens, 2003). Similar information can be gathered over a longer time period by asking users to record diary entries (Bolger, Davis, & Rafaeli, 2003; Hanington, & Martin, 2012). We can also observe users (Brill & Knauss, 2011) or use screen capture to track interactions (Kim et al., 2008).
Importantly, subjective aspects of user experience can introduce measurement obstacles. Traditional assessments rely on self-report, human-system performance, and observations that incompletely capture users’ unconscious or difficult-to-express psychological states (Podsakoff et al., 2003; Vermeeren et al., 2010), such as visual attention (Jarodzka, Scheiter, Gerjets & van Gog, 2010; Tsai, Hou, Lai, Liu, & Yang, 2012), cognitive workload (Dirican & Göktürk, 2011), and emotions (Boucsein, 1992; Cohn & De la Torre, 2014). For example, respondents’ comprehension of survey and interview questions can be flawed, resulting in responses that are incorrect or incomplete (Hanington & Martin, 2012). Similarly, constructing nonreactive and unbiased survey and interview items is a difficult process, with “loaded” and “double-barreled” questions as common problems (Lee, 2006). Even when items are otherwise clear and unbiased, many people have trouble with honest and accurate introspection (Van Gog, Paas, Van Merriënboer, & Witte, 2005)—people are not always aware of and able to articulate their actual knowledge, abilities, and emotions.
Semi-structured and open interview methods allow analysts to clarify questions and tasks, and ask probing follow-up questions, which can improve the quality of responses from users. These techniques can ameliorate but not fully solve problems of self-monitoring accuracy and respondent communication skills (Hanington & Martin, 2012). Observational methods sidestep the problems of self-report and self-disclosure (Goodman, Kuniavsky, & Moed, 2012), particularly when users are unaware of the observation (Ghergulescu & Muntean, 2014). Thus, observations of users’ behaviors or performance can reveal patterns of errors, skills, and interactions that do not depend on self-report at all. However, underlying psychological states remain hidden and must be inferred (by human analysts) from external data. Both interviews and observations also depend upon strong communication, inquiry, and record-keeping skills and training for the researchers—another source of human error and subjectivity.
In sum, capturing how users think and feel when they are using various products, services, or software remains a significant challenge for user experience evaluation. Traditional measures may fail to reflect and record users’ real-time psychological states and their impact on behavior and decision making (Mandryk et al., 2006; Yao et al., 2014). Researchers and practitioners have thus begun to explore new approaches for user experience that rely on objective biometric measures rather than solely self-report or behavioral inferences (Calvo, D'Mello, Gratch, & Kappas, 2014; Ghergulescu & Muntean, 2014; Yao et al., 2014).