Image Processing and Machine Learning Techniques for Facial Expression Recognition

Image Processing and Machine Learning Techniques for Facial Expression Recognition

Anastasios Koutlas (University of Ioannina, Greece) and Dimitrios I. Fotiadis (University of Ioannina, Greece)
DOI: 10.4018/978-1-60566-314-2.ch016
OnDemand PDF Download:


The aim of this chapter is to analyze the recent advances in image processing and machine learning techniques with respect to facial expression recognition. A comprehensive review of recently proposed methods is provided along with an analysis of the advantages and the shortcomings of existing systems. Moreover, an example for the automatic identification of basic emotions is presented: Active Shape Models are used to identify prominent features of the face; Gabor filters are used to represent facial geometry at selected locations of fiducial points and Artificial Neural Networks are used for the classification into the basic emotions (anger, surprise, fear, happiness, sadness, disgust, neutral); and finally, the future trends towards automatic facial expression recognition are described.
Chapter Preview


The face is the fundamental part of day to day interpersonal communication. Humans use the face along with facial expressions to denote consciously their emotional states (anger, surprise, stress, etc.) or subconsciously (yawn, lip biting), to accompany and enhance the meaning of their thoughts (wink) or exchange thoughts without talking (head nodes, look exchanges). Facial expressions are the result of the deformation in a human’s face due to muscle movement. The importance of automating the task to analyse facial expressions using computing systems is apparent and can be beneficial to many different scientific subjects such as psychology, neurology, psychiatry, as well as, applications of everyday life such as driver monitoring systems, automated tutoring systems or smart environments and human-computer interaction. Although humans are able to identify changes in facial expressions easily and effortlessly even in complicated scenes, the same is not an easy task to be undertaken by a machine. Moreover, computing systems must share the same robustness and accuracy with a human so that these systems could be used in a real-world scenario and provide adequate aid.

Advances in topics such as face detection, face tracking and recognition, psychological studies as well as the processing power of modern computer systems make the automatic analysis of facial expressions possible for use with real world examples where responsiveness (i.e. real time processing) is required along with sensitivity (i.e. being able to detect various day to day emotional states and visual cues) and the ability to tolerate head movements or sudden changes.

For an effective automatic facial expression recognition (AFER) system there are several characteristics that must be present so that it can be efficient. These are outlined in the Figure 1.

Figure 1.

Structure of an automatic facial expression recognition system

Face detection and identification of prominent features is a crucial step for an AFER system. It is the first step for any system that carries the automatic tag and the performance of this step in terms of accuracy is crucial for the overall accuracy of the system. Various approaches are presented in the literature in terms of static or temporal identification of the face or identification of prominent features such as eyes in contrast to identifying the presence of a face in a scene.

When the face is located it must be modeled so that it can be represented in an appropriate manner. The facial representation could be based on the facial geometry that encompasses some unique features of homogeneity and diversion across humans. It could also be based in characteristics that appear after some transformation with mathematical expressions modeling texture, position and gray-level information. After that the feature vector is built by extracting features. It can be represented either holistically or locally. Holistic approach treats the face as a whole, i.e. the processing of the face and the mathematical information applies to the whole face without considering any special prominent features of it. On the other hand the local approach treats each prominent feature of the face in a different way and the feature extraction process is applied in selected locations in the image which are often called fiducial points. Lastly, there are systems which are related to the processing of image sequences or static images which combine the two approaches, treating the face in a hybrid manner. There is also a distinction in terms of the presence of temporal information or not.

Classification is the last step for an AFER system. The facial actions or the deformations due to facial movement are categorized either as basic emotions or as Action Units (AUs). In what follows depending on the use of temporal characteristics or not the classification process is considered temporal or static for this chapter.

This chapter introduces recent advances in automatic facial expression recognition. The first part contains an introduction to the automatic facial expression recognition systems, including their structure, their objectives and their limitations. In the second part a review of recent work, is presented related to face identification, acquisition and recognition, facial feature transformation, feature vector extraction and classification. In part three a particular approach is described along with quantitative results.

Key Terms in this Chapter

Facial Action Coding System (FACS): It is a system developed by Ekman and Friesen (1978) to categorize human expressions. Using FACS human coders can categorize all possible facial deformation into action units that describe facial muscle movement.

Action Unit (AU): The key element of FACS, each action unit describes facial deformation due to each facial muscle movement. There are a total of 44 AUs where the majority involves contraction or relaxation of facial muscles and the rest involve miscellaneous actions such as “tongue show” or “bite lip”

Point Distribution Model (PDM): It is a model that tries to form a distribution of sample points from the training set. When the PDM is constructed it can approximate the position of each model point in a new image without manual intervention.

Machine Learning: The purpose of machine learning is to extract information from several types of data automatically, using computational and statistical methods. It is the use of computer algorithms which improve automatically using experience

Basic Emotions: They are a small set of prototypic emotions which share characteristics of universality and uniformity across people with different ethnic background or cultural heritage. The six basic emotions were proposed by Ekman and Friesen (1971) and are: disgust, fear, joy, surprise, sadness and anger

Classi fication: The task that categorizes feature vectors into appropriate categories. Each category is called a class.

Feature Vector Extraction: The task of providing a feature vector that describes facial geometry and deformation. There are two ways to model facial geometry and deformation, first by using prominent features of the face and second by using a mathematical transformation so that changes in appearance are modeled

Image Processing: The analysis of an image using techniques that can identify shades, colors and relationships which cannot be perceived by the human eye

Complete Chapter List

Search this Book:
Editorial Advisory Board
Table of Contents
Themis P. Exarchos, Athanasios Papadopoulos, Dimitrios I. Fotiadis
Chapter 1
Ioannis Dimou, Michalis Zervakis, David Lowe, Manolis Tsiknakis
The automation of diagnostic tools and the increasing availability of extensive medical datasets in the last decade have triggered the development... Sample PDF
Computational Methods and Tools for Decision Support in Biomedicine: An Overview of Algorithmic Challenges
Chapter 2
William Hsu, Alex A.T. Bui, Ricky K. Taira, Hooshang Kangarloo
Though an unparalleled amount and diversity of imaging and clinical data are now collected as part of routine care, this information is not... Sample PDF
Integrating Imaging and Clinical Data for Decision Support
Chapter 3
Spyretta Golemati, John Stoitsis, Konstantina S. Nikita
The estimation of motion of the myocardial and arterial wall is important for the quantification of tissue elasticity and contractility and has... Sample PDF
Analysis and Quantification of Motion within the Cardiovascular System: Implications for the Mechanical Strain of Cardiovascular Structures
Chapter 4
Christos V. Bourantas, Katerina Naka, Dimitrios Fotiadis, Lampros Michalis
Intracoronary Ultrasound (ICUS) imaging is an intravascular catheter-based technique which provides real-time, high resolution, cross-sectional... Sample PDF
New Developments in Intracoronary Ultrasound Processing
Chapter 5
Stavroula Mougiakakou, Ioannis Valavanis, Alexandra Nikita, Konstantina S. Nikita
Recent advances in computer science provide the intelligent computation tools needed to design and develop Diagnostic Support Systems (DSSs) that... Sample PDF
Diagnostic Support Systems and Computational Intelligence: Differential Diagnosis of Hepatic Lesions from Computed Tomography Images
Chapter 6
Marotesa Voultsidou, J. Michael Herrmann
Indicative features of an fMRI data set can be evaluated by methods provided by theory of random matrices (RMT). RMT considers ensembles of matrices... Sample PDF
Significance Estimation in fMRI from Random Matrices
Chapter 7
Dimitrios C. Karampinos, Robert Dawe, Konstantinos Arfanakis, John G. Georgiadis
Diffusion Magnetic Resonance Imaging (diffusion MRI) can provide important information about tissue microstructure by probing the diffusion of water... Sample PDF
Optimal Diffusion Encoding Strategies for Fiber Mapping in Diffusion MRI
Chapter 8
Dimitrios G. Tsalikakis, Petros S. Karvelis, Dimitrios I. Fotiadis
Segmentation plays a crucial role in cardiac magnetic resonance imaging (CMRI) applications, since it permits automated detection of regions of... Sample PDF
Segmentation of Cardiac Magnetic Resonance Images
Chapter 9
Katia Marina Passera, Luca Tommaso Mainardi
Image registration is the process of determining the correspondence of features between images collected at different times or using different... Sample PDF
Image Registration Algorithms for Applications in Oncology
Chapter 10
Lena Costaridou, Spyros Skiadopoulos, Anna Karahaliou, Nikolaos Arikidis, George Panayiotakis
Breast cancer is the most common cancer in women worldwide. Mammography is currently the most effective modality in detecting breast cancer... Sample PDF
Computer-Aided Diagnosis in Breast Imaging: Trends and Challenges
Chapter 11
E. Kyriacou, C.I. Christodoulou, C. Loizou, M.S. Pattichis, C.S. Pattichis, S. Kakkos
Stroke is the third leading cause of death in the Western world and a major cause of disability in adults. The objective of this work was to... Sample PDF
Assessment of Stroke by Analysing Cartoid Plaque Morphology
Chapter 12
Marios Neofytou, Constantinos Pattichis, Vasilios Tanos, Marios Pattichis, Eftyvoulos Kyriacou
The objective of this chapter is to propose a quantitative hysteroscopy imaging analysis system in gynaecological cancer and to provide the current... Sample PDF
Quantitative Analysis of Hysteroscopy Imaging in Gynecological Cancer
Chapter 13
Thomas V. Kilindris, Kiki Theodorou
Patient anatomy, biochemical response, as well functional evaluation at organ level, are key fields that produce a significant amount of multi modal... Sample PDF
Combining Geometry and Image in Biomedical Systems: The RT TPS Case
Chapter 14
Ioannis Tsougos, George Loudos, Panagiotis Georgoulias, Konstantina S. Nikita, Kiki Theodorou
Quantitative three-dimensional nuclear medical imaging plays a continuously increasing role in radionuclide dosimetry, allowing the development of... Sample PDF
Internal Radionuclide Dosimetry using Quantitative 3-D Nuclear Medical Imaging
Chapter 15
Evanthia E. Tripoliti, Dimitrios I. Fotiadis, Konstantia Veliou
Diffusion Tensor Imaging (DTI) is a magnetic resonance imaging (MRI) modality which can significantly improve our understanding of the brain... Sample PDF
Diffusion Tensor Imaging and Fiber Tractography
Chapter 16
Anastasios Koutlas, Dimitrios I. Fotiadis
The aim of this chapter is to analyze the recent advances in image processing and machine learning techniques with respect to facial expression... Sample PDF
Image Processing and Machine Learning Techniques for Facial Expression Recognition
Chapter 17
Arcangelo Merla
This chapter presents an overview on recent developments in the field of clinical applications of the functional infrared imaging. The functional... Sample PDF
Developments and Advances in Biomedical Functional Infrared Imaging
Chapter 18
Aristotelis Chatziioannou, Panagiotis Moulos
The completion of the Human Genome Project and the emergence of high-throughput technologies at the dawn of the new millennium, are rapidly changing... Sample PDF
DNA Microarrays: Analysis and Interpretation
Chapter 19
Nikolaos Giannakeas, Dimitrios I. Fotiadis
Microarray technology allows the comprehensive measurement of the expression level of many genes simultaneously on a common substrate. Typical... Sample PDF
Image Processing and Machine Learning Techniques for the Segmentation of cDNA
Chapter 20
Petros S. Karvelis, Dimitrios I. Fotiadis
Automated chromosome analysis is now becoming routine in most human cytogenetics laboratories. It involves both processing and analysis of digital... Sample PDF
Recent Advances in Automated Chromosome Image Analysis
Chapter 21
O. Lezoray, G. Lebrun, C. Meurie, C. Charrier, A. Elmotataz, M. Lecluse
The segmentation of microscopic images is a challenging application that can have numerous applications ranging from prognosis to diagnosis.... Sample PDF
Machine Learning in Morphological Segmentation
Chapter 22
Michael Haefner, Alfred Gangl, Michael Liedlgruber, A. Uhl, Andreas Vecsei, Friedrich Wrba
Wavelet-, Fourier-, and spatial domain-based texture classification methods have been used successfully for classifying zoom-endoscopic colon images... Sample PDF
Pit Pattern Classification Using Multichannel Features and Multiclassification
Chapter 23
C. Papaodysseus, P. Rousopoulos, D. Arabadjis, M. Panagopoulos, P. Loumou
In this chapter the state of the art is presented in the domain of automatic identification and classification of bodies on the basis of their... Sample PDF
Automatic Identification and Elastic Properties of Deformed Objects Using their Microscopic Images
Chapter 24
Alexia Giannoula, Richard S.C. Cobbold
“Elastography” or “elasticity imaging” can be defined as the science and methodology of estimating the mechanical properties of a medium (including... Sample PDF
Nonlinear Ultrasound Radiation-Force Elastography
Chapter 25
Valentina Russo, Roberto Setola
The aim of this chapter is to provide an overview about models and methodologies used for the Dynamic Contrast Enhancement (DCE) analysis. DCE is a... Sample PDF
Dynamic Contrast Enhancement: Analysis's Models and Methodologies
Chapter 26
George K. Matsopoulos
The accurate estimation of point correspondences is often required in a wide variety of medical image processing applications including image... Sample PDF
Automatic Correspondence Methods towards Point-Based Medical Image Registration: An Evaluation Study
Chapter 27
Alberto Taboada-Crispi, Hichem Sahli, Denis Hernandez-Pacheco, Alexander Falcon-Ruiz
Various approaches have been taken to detect anomalies, with certain particularities in the medical image scenario, linked to other terms... Sample PDF
Anomaly Detection in Medical Image Analysis
Chapter 28
C. Delgorge-Rosenberger, C. Rosenberger
The authors present in this chapter an overview on evaluation of medical image compression. The different methodologies used in the literature are... Sample PDF
Evaluation of Medical Image Compression
Chapter 29
Charalampos Doukas, Ilias Maglogiannis
Medical images are often characterized by high complexity and consist of high resolution image files, introducing thus several issues regarding... Sample PDF
Advanced ROI Coding Techniques for Medical Imaging
Chapter 30
Farhang Sahba
Ultrasound imaging now has widespread clinical use. It involves exposing a part of the body to highfrequency sound waves in order to generate images... Sample PDF
Segmentation Methods in Ultrasound Images
About the Editors
About the Contributors