Abstract
We seem to be entering an era of enhanced digital connectivity. Computers and Internet have become so embedded in the daily fabric of people’s lives that people simply cannot live without them (Hoffman, Novak, & Venkatesh, 2004). We use this technology to work, to communicate, to shop, to seek out new information, and to entertain ourselves. With this ever-increasing diffusion of computers in society, human–computer interaction (HCI) is becoming increasingly essential to our daily lives. HCI design was first dominated by direct manipulation and then delegation. The tacit assumption of both styles of interaction has been that the human will be explicit, unambiguous, and fully attentive while controlling the information and command flow. Boredom, preoccupation, and stress are unthinkable even though they are “very human” behaviors. This insensitivity of current HCI designs is fine for well-codified tasks. It works for making plane reservations, buying and selling stocks, and, as a matter of fact, almost everything we do with computers today. But this kind of categorical computing is inappropriate for design, debate, and deliberation. In fact, it is the major impediment to having flexible machines capable of adapting to their users and their level of attention, preferences, moods, and intentions. The ability to detect and understand affective states of a person we are communicating with is the core of emotional intelligence. Emotional intelligence (EQ) is a facet of human intelligence that has been argued to be indispensable and even the most important for a successful social life (Goleman, 1995). When it comes to computers, however, not all of them will need emotional intelligence and none will need all of the related skills that we need. Yet human–machine interactive systems capable of sensing stress, inattention, and heedfulness, and capable of adapting and responding appropriately to these affective states of the user are likely to be perceived as more natural, more efficacious, and more trustworthy. The research area of machine analysis of human affective states and employment of this information to build more natural, flexible (affective) HCI goes by a general name of affective computing, introduced first by Picard (1997).
TopResearch Motivation
Besides the research on natural, flexible HCI, various research areas and technologies would benefit from efforts to model human perception of affective feedback computationally. For instance, automatic recognition of human affective states is an important research topic for video surveillance as well. Automatic assessment of boredom, inattention, and stress will be highly valuable in situations where firm attention to a crucial, but perhaps tedious task is essential, such as aircraft control, air traffic control, nuclear power plant surveillance, or simply driving a ground vehicle like a truck, train, or car. An automated tool could provide prompts for better performance based on the sensed user’s affective states.
Another area that would benefit from efforts towards computer analysis of human affective feedback is the automatic affect-based indexing of digital visual material. A mechanism for detecting scenes/frames which contain expressions of pain, rage, and fear could provide a valuable tool for violent-content-based indexing of movies, video material and digital libraries.
Other areas where machine tools for analysis of human affective feedback could expand and enhance research and applications include specialized areas in professional and scientific sectors. Monitoring and interpreting affective behavioral cues are important to lawyers, police, and security agents who are often interested in issues concerning deception and attitude. Machine analysis of human affective states could be of considerable value in these situations where only informal interpretations are now used. It would also facile the research in areas such as behavioral science (in studies on emotion and cognition), anthropology (in studies on cross-cultural perception and production of affective states), neurology (in studies on dependence between emotional abilities impairments and brain lesions), and psychiatry (in studies on schizophrenia) in which reliability, sensitivity, and precision are persisting problems. For a further discussion, see Pantic and Bartlett (2007) and Pantic, Pentland, Nijholt, and Huang (2007).
Key Terms in this Chapter
Affective Computing: The research area concerned with computing that relates to, arises from, or deliberately influences emotion. Affective computing expands HCI by including emotional communication together with appropriate means of handling affective information.
Context-sensitive HCI: HCI in which the computer’s context with respect to nearby humans (i.e., who the current user is, where he is, what his current task is, and how he feels) is automatically sensed, interpreted, and used to enable the computer to act or respond appropriately.
Anticipatory Interface: Software application that realizes human–computer interaction by means of understanding and proactively reacting (ideally, in a context-sensitive manner) to certain human behaviors such as moods and affective feedback.
Human–Computer Interface: A software application, a system that realizes human-computer interaction.
Multimodal (Natural) HCI: HCI in which command and information flow exchanges via multiple natural sensory modes of sight, sound, and touch. The user commands are issued by means of speech, hand gestures, gaze direction, facial expressions, and so forth, and the requested information or the computer’s feedback is provided by means of animated characters and appropriate media.
Emotional Intelligence: A facet of human intelligence that includes the ability to have, express, recognize, and regulate affective states, employ them for constructive purposes, and skillfully handle the affective arousal of others. The skills of emotional intelligence have been argued to be a better predictor than IQ for measuring aspects of success in life.
Human–Computer Interaction (HCI): The command and information flow that streams between the user and the computer. It is usually characterized in terms of speed, reliability, consistency, portability, naturalness, and users’ subjective satisfaction.