Advances in Face Image Analysis: Techniques and Technologies
Book Citation Index

Advances in Face Image Analysis: Techniques and Technologies

Yu-Jin Zhang (Tsinghua University, China)
Indexed In: SCOPUS View 3 More Indices
Release Date: July, 2010|Copyright: © 2011 |Pages: 404
ISBN13: 9781615209910|ISBN10: 1615209913|EISBN13: 9781615209927|DOI: 10.4018/978-1-61520-991-0


Video and image analysis of the human face provides a wealth of information about the individual, , including age, behavior, health and profession. With research continually being conducted into multiple applications of this field, a comprehensive and detailed volume of the new advancements of face image analysis is in demand.

Advances in Face Image Analysis: Techniques and Technologies fulfills this need, reviewing and surveying new forward-thinking research and development in face image analysis technologies. With more than 30 leading experts from around the world providing comprehensive coverage of various branches of face image analysis, this book is a valuable asset for students, researchers and practitioners engaged in the study, research and development of face image analysis techniques.

Topics Covered

The many academic areas covered in this publication include, but are not limited to:

  • Configural Processing Hypothesis and Face-Inversion Effect
  • Discriminant Learning Using Training Space Partitioning
  • Facial Expression Analysis by Machine Learning
  • Facial Feature Detection Algorithms
  • Feature Selection in High Dimension
  • Gabor and Log-Gabor Wavelet for Face Recognition
  • Invariance with Compound Image Transforms
  • Photometric Normalization for Illumination Invariance
  • Subtle Facial Expression Recognition
  • Transform Based Feature Extraction

Reviews and Testimonials

To push the deep theoretical study of this important area, to prompt the further development of face analysis techniques, and to guide the people working in image area for applying suitable methods, a comprehensive and detailed volume for the new advancements of face image analysis would be a must, and these are the overall objectives and the main mission of this book. Comprehensive coverage of various branches of face image analysis is provided by more than 30 leading experts from 16 countries and regions around the world.

– Yu-Jin Zhang, Tsinghua University, China

Table of Contents and List of Contributors

Search this Book:


The analysis of the human face via image (and video) is one of the most interesting and focusing research topics in the last years for the image community. From the analysis (sensing, computing, and perception) of face images, much information can be extracted, such as the sex/gender, age, facial expression, emotion/temper, mentality/mental processes and behavior/psychology, and the health of the person captured. According to this information, many practical tasks can be performed and completed; these include not only person identification or verification (face recognition), but also the estimation and/or determination of person’s profession, hobby, name (recovered from memory), etc.

Research on face image analysis has been carried out and is being conducted around various application topics, such as (in alphabetical order) age estimation, biometrics, biomedical instrumentations, emotion assessment, face recognition, facial expression classification, gender determination, human-computer/human-machine interaction, human behavior and emotion study, industrial automation, military service, psychosis judgment, security checking systems, social signal processing, surveillance systems, sport training, tele-medicine service, etc.

To push the deep theoretical study of this important area, to prompt the further development of face analysis techniques, and to guide the people working in image area for applying suitable methods, a comprehensive and detailed volume for the new advancements of face image analysis would be a must, and these are the overall objectives and the main mission of this book. Comprehensive coverage of various branches of face image analysis is provided by more than 30 leading experts from 16 countries and regions around the world.

In addition, new achievements and new challenges often come up together. With the accumulation of research results in face image analysis, the need for a uniform description of this area and a review of recent developments are even increasing. This book tries to set the fundamental knowledge for, and introduce the advancements in recent years in, face image analysis. The principal concern of this book is to provide those in the face image analysis community with a general methodology that allows them to integrate the research results and to apply them in various applications.

The complete coverage of this book is also reflected in the title, “Advances in Face Image Analysis: Techniques and Technologies,” where techniques would refer to the different methodologies and models utilized, whereas technologies would refer to the applications and tools applied. This book is intended for scientists and engineers who are engaged in the research and development of face image analysis techniques and who wish to keep their paces with the advances of this field. The objective of this collection is to review and survey new forward-thinking research and development in face image analysis technologies.

Face image analysis is a specialized field of image analysis (that is an important layer of image engineering), though face images have some particularities, the methodologies and techniques presented in this book are ready to be extended to other types of images, and are the foundation of many image analysis applications.

As various face analysis techniques have been widely used in many areas, the audience of such a book should be all persons that study/research in the disciplines such as, image and information processing, computer vision, pattern recognition. Such a book would be suitable for using in specialized courses for graduate students and to be read by researchers in companies of related domains.

The whole book includes 17 chapters and they are organized into six sections. They covers several distinctive research fields in face image analysis and form a solid background for treating feature detection, face recognition, facial expression, etc. They also provide many state-of-the-art advancements and achievements in this field. Some detailed descriptions, respectively for each section and each chapter, are provided in the following.

Section I is an “Introduction and Background” section, which is for the purpose of providing some historical surroundings information, various achieved results and a brief overview of current research focus of face image analysis, which consists of one opening chapter (Chapter 1) and one surveying chapter (Chapter 2).

Chapter 1
is entitled “Face, Image, and Analysis.” The 3 terms form a unique composition, face image analysis (FIA). Face image analysis consists of automatic investigation of images of (human) faces, is a hot research topic and a fruitful field. This introductory chapter discusses several aspects of the history and scope of face image analysis and provides an outline of research development publications of this domain. More prominently, different modules and some typical techniques for face image analysis are listed, explained, described, or summarized from a general technical point of view. One picture of the advancements and the front of this complex and prominent field is provided. Finally, several challenges and prominent development directions for the future are identified.

Chapter 2 is entitled “Face Searching in Large Databases.” With the significant progress of biometric research, increasingly efficient techniques have been developed. Among them, face recognition represents a good solution for local/remote access security even under less controlled conditions. In the last decade, many algorithms based on linear/non-linear methods, neural networks, wavelets, etc. have been proposed. Nevertheless, during Face Recognition Vendor Test 2002 most of them encountered problems outdoors. This lowers the reliability of human face identification compared to other biometrics, and underlines the need for more research. This chapter provides a survey of recent outcomes on the topic, and supply a starting reference to potential investigators. Both 2-D imagery and 3-D models are addressed. Several tables containing different collections of parameters (such as input size, recognition rate, number of addressed problems) make the comparison of different methods without difficulty. Some future directions are finally proposed.

Section II is for “Facial Feature Extraction”, which is a fundamental task in all analysis problems. Once the faces from images are captured, some suitable features should be selected and extracted from these images to represent and describe the face characteristics. These features should represent and/or reflect the particularity of faces in images and provide the base for further treatments. Commonly used features include geometric features, such as distance, angle, curvature, etc; and grey level feature, such as color, texture of faces. Features can also be classified as local ones or holistic ones. This section consists of three chapters (Chapter 3 to Chapter 5) utilizing different techniques.

Chapter 3 is entitled “A Review of Facial Feature Detection Algorithms.” Facial feature detection is an important task in numerous applications of face image analysis that include face recognition and verification, facial expression recognition, driver's alertness estimation, head pose estimation etc. Thus, the area has been a very active research field for many years and a multitude of methods appear in the literature. Depending on the targeted application, the proposed methods have different characteristics and are designed to perform in different setups. Thus, a method of general applicability seems to be away from the current state of the art. This chapter intends to offer an up-to-date literature review of facial feature detection algorithms. A review of the image databases and performance metrics that are used to benchmark these algorithms is also provided.

Chapter 4 is entitled “Gabor and Log-Gabor Wavelet for Face Recognition.” Gabor wavelet is often applied to extract relevant features from a facial image in practice. This wavelet is constructed using filters of multiple scales and orientations. Based on Gabor’s theory of communication, Gabor wavelet and Log-Gabor wavelet are proposed to acquire initial features from 2-D images. Theoretically the main difference between these two wavelets is Log-Gabor wavelet produces DC free filter responses, whereas Gabor filter responses retain DC components. This experimental study determines the characteristics of Gabor and Log-Gabor filters for face recognition. In the experiment, two sixth order data tensor are created; one containing the basic Gabor feature vectors and the other containing the basic Log-Gabor feature vectors. This study reveals the characteristics of the filter orientations for Gabor and Log-Gabor filters for face recognition. These two implementations show that the Gabor filter having orientation zero means oriented at 0 degree with respect to the aligned face has the highest discriminating ability, while Log-Gabor filter with 45 degree of orientation has the highest discriminating ability. This result is consistent across three different frequencies (scales) used for this experiment. It is also observed that for both the wavelets, filters with low frequency have higher discriminating ability.

Chapter 5 is entitled “Efficient Face Retrieval Based on Bag of Facial Features.” In this chapter, an efficient retrieval technique for human face images based on bag of facial features is presented. A visual vocabulary is built beforehand using an invariant descriptor computed on detected image regions. The vocabulary is refined in two ways to make the retrieval system more efficient. Firstly, the visual vocabulary is minimized by only using facial features selected on face regions which are detected by an accurate face detector. Secondly, three criteria, namely Inverted-Occurrence-Frequency Weights, Average Feature Location Distance and Reliable Nearest-Neighbors, are calculated in advance to make the on-line retrieval procedure more efficient and precise. The proposed system is experimented on the Caltech Human Face Database. The results show that this technique is very effective and efficient on face image retrieval.

Section III is for “Feature Dimensionality Reduction”. Often a large number of features can be extracted from face images, as human face is a non-rigid natural object and many factors influence the appearance of face in images. The original features extracted correspond to the data in high dimensional space. Directly using these high dimensional data cause not only computational problems but also inaccuracy of analysis results. Therefore, making the reduction of feature dimension, before further treatments is necessary. This section consists of three chapters (Chapter 6 to Chapter 8) describing different methods for this purpose.

Chapter 6 is entitled “Feature Selection in High Dimension.” Variable selection for classification is a crucial paradigm in image analysis. Indeed, images are generally described by a large amount of features (pixels, edges …) although it is difficult to obtain a sufficiently large number of samples to draw reliable inference for classifications using the whole number of features. In this chapter, some simple and effective features selection methods based on filter strategy are described. Some more sophisticated methods based on margin criterion or stochastic approximation techniques that achieve great performances of classification with a very small proportion of variables are provided. Most of these "wrapper" methods are dedicated to a special case of classifier, except the Optimal Features Weighting (OFW) algorithm that is a meta-algorithm and works with any classifier. A large part of this chapter will be dedicated to the description of the description of OFW algorithm and hybrid OFW algorithms.

Chapter 7 is entitled “Transform Based Feature Extraction and Dimensionality Reduction.” Low dimensional feature representation with enhanced discrimination power is of paramount importance to face recognition system. This chapter presents transform based techniques for extraction of efficient and effective features to solve some of the challenges in face recognition. The techniques are based on the combination of Radon transform, Discrete Cosine Transform (DCT), and Discrete Wavelet Transform (DWT). The property of Radon transform to enhance the low frequency components, which are useful for face recognition, has been exploited to derive the effective facial features. The comparative study of various transform based techniques under different conditions like varying illumination, changing facial expressions, and in-plane rotation is presented. The experimental results using FERET, ORL, and Yale databases are also presented.

Chapter 8 is entitled “FastNMF: Efficient NMF Algorithm for Reducing Feature Dimension.” Non-negative matrix factorization (NMF) is a more and more popular method for non-negative dimensionality reduction and feature extraction of non-negative data, especially face images. Currently, no single NMF algorithm holds not only satisfactory efficiency for dimensionality reduction and feature extraction of face images but also high ease of use. To improve the applicability of NMF, this chapter proposes a new monotonic, fixed-point algorithm called FastNMF by implementing least squares error-based non-negative factorization essentially according to the basic properties of parabola functions. The minimization problem corresponding to an operation in FastNMF can be analytically solved just by this operation, which is far beyond existing NMF algorithms’ power. Therefore, FastNMF holds much higher efficiency, which is validated by a set of experimental results. From the point of view of simplicity in design philosophy, FastNMF is also one of NMF algorithms that are the easiest to use and the most comprehensible. Besides, theoretical analysis and experimental results show that FastNMF tends to extract facial features with better representation ability than popular multiplicative update-based algorithms.

Section IV is for “Face Recognition”. Based on the dimensionality reduced features, further treatment would make the recognition of the identity of faces possible. Face recognition is currently the most prominent direction in face image analysis, with both a large numbers of research publications and a various types of application systems, for both verification and identification. This section consists of three chapters (Chapter 9 to Chapter 11) concentrated on employing newly developed theories and tools for face recognition.

Chapter 9 is entitled “Sparse Representation for View-based Face Recognition.” In this chapter, the problem of face recognition using sparse representation classification (SRC) is discussed. The SRC classifier has recently emerged as one of the latest paradigm in the context of view-based face recognition. The main aim of the chapter is to provide an insight of the SRC algorithm with thorough discussion of the underlying “Compressive Sensing” theory. Comprehensive experimental evaluation of the approach is conducted on a number of standard databases using exemplary evaluation protocols to provide a comparative index with the benchmark for face recognition algorithms. The algorithm is also extended to the problem of video-based face recognition for more realistic applications.

Chapter 10 is entitled “Probabilistic Methods for Face Registration and Recognition.” In this chapter many probabilistic approaches to face recognition are reviewed and the extended treatment of one particular approach is presented. Here, the face image is decomposed into an additive sum of two parts: a deterministic component that depends on an underlying representation of identity and a stochastic component that explains the difference of two face images from the same person. Inferences about matching are made by comparing different probabilistic models rather than comparing distance to an identity template in some projected space. It is demonstrated that this model comparison is superior to distance comparison. Furthermore, its performance can be further improved by sampling the feature space and combining models trained using these feature subspaces. Both random sampling with and without replacement significantly improves performance. Finally, this probabilistic approach can be adapted for keypoint localization. The recognition and keypoint localization performance are comparable to those using manual labelling.

Chapter 11 is entitled “Discriminant Learning Using Training Space Partitioning.” Using large training databases often introduces a level of complexity that often degrades the classification performance of face recognition methods. In this chapter, an overview of various approaches that are employed in order to overcome this problem is presented first. In addition, a specific discriminant learning approach that combines dynamic training and partitioning is described in detail. This face recognition methodology employs dynamic training in order to implement a person-specific iterative classification process. This process employs discriminant clustering by making use of an entropy-based measure; the algorithm adapts the coordinates of the discriminant space with respect to the characteristics of the test face. As a result, the training space is dynamically reduced to smaller spaces, where linear separation of the face classes is more likely to be achieved. The process iterates until one final cluster is retained, which consists of a single face class that represents the best match to the test face. The performance of this methodology is evaluated on standard face databases and results show that the proposed framework gives a good solution to the face recognition problem.

Section V is for “Facial Expression Classification”. This task has long time attracted the attention of sociologists and become a focused theme recently in engineering community. Different classes of expression have been defined. The sociologists often classify facial expression into 18 groups, while many computer scientists follow the classification scheme of psychoanalyst to classify facial expression into 6 basic kinds. Another classification scheme combining also the knowledge from anatomy uses 44 action units to describe various facial expressions. This section consists of three chapters (Chapter 12 to Chapter 14) explaining some recent advancements.

Chapter 12 is entitled “From Face to Facial Expression.” This chapter addresses recent advances in computer vision for facial expression classification. The different processing steps of the problem of automatic facial expression recognition are presented. The advances of each stage of the problem are described and the future challenges towards the application of such systems to every day life situations are discussed. The importance of taking advantage of the human strategy by reviewing advances of research in psychology towards multidisciplinary approach for facial expression classification is also introduced. Finally, one contribution that aims at dealing with some of the discussed challenges is provided.

Chapter 13 is entitled “Facial Expression Analysis by Machine Learning.” Considering a facial expression is formed by contracting or relaxing different facial muscles on human face that results in temporally deformed facial features, some challenges of such system are discussed. For instances, lighting condition is a very difficult problem to constraint and regulate. On the other hand, real-time processing is also a challenging problem since there are so many facial features to be extracted and processed. Often, conventional classifiers are not even effective to handle those features and then produce good classification performance. This chapter discusses the issues on how the advanced feature selection techniques together with good classifiers can play a vital important role of real-time facial expression recognition. Several feature selection methods and classifiers are discussed and their evaluations for real-time facial expression recognition are presented. This chapter opens-up a discussion about building a real-time system to read and respond to the emotions of people from facial expressions.

Chapter 14 is entitled “Subtle Facial Expression Recognition in Still Images and Videos.” This chapter addresses the recognition of basic facial expressions. It has three main contributions. First, a view- and texture independent schemes that exploits facial action parameters estimated by an appearance-based 3D face tracker is introduced. The learned facial actions associated with different facial expressions by time series are represented. Two dynamic recognition schemes are proposed: one is based on conditional predictive models and on an analysis-synthesis scheme; another is based on examples allowing straightforward use of machine learning approaches. Second, an efficient recognition scheme based on the detection of key-frames in videos is proposed. Third, the dynamic scheme with a static one based on analyzing individual snapshots is compared, the result shows that in general the former performs better than the latter. Evaluations of performance using Linear Discriminant Analysis (LDA), Non parametric Discriminant Analysis (NDA), and Support Vector Machines (SVM) are provided.

Section IV deals with some “Invariance Techniques”. Many techniques have been developed for face images obtained under frontal pose and optimal lighting conditions, though their performances attend reasonable high score, the recognition performance severely degrades with pose and lighting variations, as well as expression variations. These variations, for example, could make one’s face image more like other person’s images than his/her own image. This section consists of three chapters (Chapter 15 to Chapter 17) presenting suitable techniques for cope with these variations for face recognition.

Chapter 15 is entitled “Photometric Normalization Techniques for Illumination Invariance.” Thanks to the enormous research effort made by different research groups from universities and companies around the world, the recognition rates achievable with the state-of-the-art face recognition technology are steadily growing, even though some issues still pose major challenges to the technology. Amongst these challenges, coping with illumination-induced appearance variations is one of the biggest and still not satisfactorily solved. A number of techniques have been proposed in the literature to cope with the impact of illumination ranging from simple image enhancement techniques, such as histogram equalization, to more elaborate methods, such as anisotropic smoothing or the logarithmic total variation model. This chapter presents an overview of the most popular and efficient normalization techniques that try to solve the illumination variation problem at the preprocessing level. It assesses the techniques on the YaleB and XM2VTS databases and explores their strengths and weaknesses from the theoretical and implementation point of view.

Chapter 16 is entitled “Pose and Illumination Invariance with Compound Image Transforms.” While current face recognition algorithms have provided convincing performance on frontal face poses, recognition is far less effective when the pose and illumination conditions vary. This chapter shows how compound image transforms can be used for face recognition in various poses and illumination conditions. The method works by first dividing each image into four equal-sized tiles. Then, image features are extracted from the face images, transforms of the images, and transforms of transforms of the images. Finally, each image feature is assigned with a Fisher score, and test images are classified by using a simple Weighted Nearest Neighbor rule such that the Fisher scores are used as weights. Experimental results using the full color FERET dataset show that with no parameter tuning, the accuracy of rank-10 recognition for frontal, quarter-profile, and half-profile images is ~98%, ~94% and ~91%, respectively. The proposed method also achieves perfect accuracy on several other face recognition datasets such as Yale B, ORL and JAFFE. An important feature of this method is that the recognition accuracy improves as the number of subjects in the dataset gets larger.

Chapter 17 is entitled “Configural Processing Hypothesis and Face-Inversion Effect.” Perception and recognition of faces presented upright are better than that of faces presented inverted. The difference between upright and inverted orientations is greater in face recognition than in non-face object recognition. This face-inversion effect is explained by the "Configural Processing" hypothesis that inversion disrupts configural information processing and leaves the featural information intact. This chapter discusses two important findings that cast doubt on this hypothesis: inversion impairs recognition of isolated features (hair & forehead, and eyes), and certain facial configural information is not affected by inversion. The chapter focuses mainly on the latter finding, which reveals a new type of facial configural information, the "Eye-Illusion", which is based on certain geometrical illusions. The eye-illusion tended to resist inversion in experimental tasks of both perception and recognition. It resisted inversion also when its magnitude was reduced. Similar results were obtained with "Headlight-Illusion" produced on a car's front, and with "Form-Illusion" produced in geometrical forms. However, the eye-illusion was greater than the headlight-illusion, which in turn was greater than the form-illusion. These findings were explained by the "General Visual-Mechanism" hypothesis in terms of levels of visual information learning. The chapter proposes that a face is composed of various kinds of configural information that are differently impaired by inversion: from no effect (the eye-illusion) to a large effect (the face-inversion effect).

In summary, the 17 chapters in 6 sections, a total of 148 figures, 54 tables, 243 equations, 124 key terms and definition, and the list of more than 800 cited references of this book offer a comprehensive and vivid image about the recent achievements and current state of face image analysis, and provide a set of general and detailed information on the advances in face image analysis.

Author(s)/Editor(s) Biography

Yu-Jin Zhang, PhD, State University of Liège, Belgium, is Professor of Image Engineering at Tsinghua University, Beijing, China. Previously, he was with the Delft University of Technology, the Netherlands. In 2003, he spent the sabbatical year as visiting professor at National Technological University, Singapore. His research interests are mainly in the area of image engineering, including image processing, image analysis and image understanding, as well as their applications. He has published nearly 400 research papers and more than 20 books, including three monographs: Image Segmentation (Science Press, 2001), Content-based Visual Information Retrieval (Science Press, 2003) and Subspace-based Face Recognition (Tsinghua University Press, 2009), as well as two edited collections: Advances in Image and Video Segmentation (IRM Press, 2006) and Semantic-based Visual Information Retrieval (IRM Press, 2007). He is vice president of China Society of Image and Graphics and the director of the academic committee of the society. He is deputy editor-in-chief of Journal of Image and Graphics and at the editorial board of several international journals. He was the program co-chair of The First, Second, Fourth, and Fifth International Conference on Image and Graphics (ICIG'2000, ICIG'2002, ICIG'2007, and ICIG'2009). He is a senior member of IEEE.