Facial Expression Analysis by Machine Learning

Introduction

Given the significant role of the face in our emotional and social lives, it is not surprising that the potential benefits from efforts to automate the analysis of facial signals, in particular rapid facial signals, are varied and numerous (Ekman et al., 1993), especially when it comes to computer science and technologies brought to bear on these issues (Pantic, 2006). As far as natural interfaces between humans and computers are concerned, facial expressions provide a way to communicate basic information about needs and demands to the machine. In fact, automatic analysis of facial signals seem to have a natural place in various vision sub-systems, including automated tools for tracking gaze and focus of attention, lip reading, bimodal speech processing, face/visual speech synthesis, and face-based command issuing.

Facial Expression Analysis is a challenging task. A facial expression is formed by contracting or relaxing different facial muscles on human face that results in temporally deformed facial features like wide-open mouth, raising eyebrows or etc. The challenges of such system have to address the following issues:

Lighting conditions is a very difficult problem to constraint and regulate. The strength of the light depends on the light source (see Figure 1).

Figure 1.

Light variations problem: face images are taken from different illumination conditions (source: Yale Face Database B http://cvc.yale.edu/projects/yalefacesB/yalefacesB.html)

The direction of the subjects face is not always ideal which may pose difficulties when the system is implemented live that captures moving subjects’ facial expression (see Figure 2).

Figure 2.

Pose variations problem: face images are taken from different postures of the object (source NTU Asian Emotion Database http://www3.ntu.edu.sg/SCE/labs/forse/Asian%20Emotion%20Database.htm)

Another difficulty is the way image is acquired by the image acquisition system. The characteristics of the image acquisition system can affect the quality of the images or videos captured.

Occlusion of subject face may tumble the hit rate of many established approaches. The experiments being carried out by most researchers do not take occlusion into account (see Figure 3).

Figure 3.

Occlusion problems: facial components are occluded by some artifact objects. (Source NTU Asian Emotion Database http://www3.ntu.edu.sg/SCE/labs/forse/Asian%20Emotion%20Database.htm)