The objective of this chapter is to introduce the reader to the recent advances in computer processing of facial expressions and communicated affect. Human facial expressions have evolved in tandem with human face recognition abilities, and show remarkable consistency across cultures. Consequently, it is rewarding to review the main traits of face recognition in humans, as well as consolidated research on the categorization of facial expressions. The bulk of the chapter focuses on the main trends in computer analysis of facial expressions, sketching out the main algorithms and exposing computational considerations for different settings. The authors then look at some recent applications and promising new projects to give the reader a realistic view of what to expect from this technology now and in near future.
TopCategorization Of Facial Expressions
The human face is a complicated visual object; it contains a lot of information with regards to identity, communicative intent and affect, and humans can “read” these cues, even under difficult visibility conditions. We can for instance understand the emotions of a person we see for the first time. In this section we look at taxonomies of facial expressions, and point out to several important factors that need to be taken into account in evaluating facial expressions.
A facial expression can be the result of an emotional response (spontaneous), or a construct with communicative intent (volitional) (Russell & Fernandez–Dols, 1997). It can occur naturally, or it can be posed. In both cases, it can have different intensities, and it can be a mixture of pure expressions. These factors make the task of sorting out a facial expression difficult. Additionally, the categorization of expressions can be achieved in ever-finer levels. It is one thing to label the category of an expression as “happy”, quite another to distinguish between a real smile (also called a Duchenne smile), a miserable smile, an angry smile, an embarrassed smile, and a dimpler. Finally, cultural differences in facial expressions also need to be taken into account.
Categorization of emotions predate computers by hundreds of years, but the roles of particular emotions in society are different for each culture; in India, for instance, it was believed that the basic emotions are sexual passion, anger, disgust, perseverance, amusement, sorrow, wonder, fear, and serenity. Facial expressions of these emotions are culture-dependent, but also the semantic counterparts of these emotions do not completely overlap with the current understanding of these words, adding to the difficulty of systematically categorizing emotions. Furthermore, the experimental settings under which any study is conducted and the ensuing databases on which we measure the success of a given method are not independent of cultural influences. For instance it is known that in some cultures the expression of emotion is more restricted for social reasons. Finally, as facial morphology also changes according to the anthropological group of a subject, it is natural to expect some principled variation across races and gender.