In this chapter, the authors present some issues related to automatic face image tagging techniques. Their main purpose in user applications is to support the organization (indexing) and retrieval (or easy browsing) of images or videos in large collections. Their core modules include algorithms and strategies for handling very large face databases, mostly acquired in real conditions. As a background for understanding how automatic face tagging works, an overview about face recognition techniques is given, including both traditional approaches and novel proposed techniques for face recognition in uncontrolled settings. Moreover, some applications and the way they work are summarized, in order to depict the state of the art in this area of face recognition research. Actually, many of them are used to tag faces and to organize photo albums with respect to the person(s) presented in annotated photos. This kind of activity has recently expanded from personal devices to social networks, and can also significantly support more demanding tasks, such as automatic handling of large editorial collections for magazine publishing and archiving. Finally, a number of approaches to large-scale face datasets as well as some automatic face image tagging techniques are presented and compared. The authors show that many approaches, both in commercial and research applications, still provide only a semi-automatic solution for this problem.
TopIntroduction
Face image tagging is the process of assigning an identification code (e.g. a personal name) to a face image according to the depicted persons, and is traditionally made by hand. This supports both indexing/classification and retrieval operations. In general, the task is not that demanding when it pertains to personal collections, e.g., the photos of participants in a social network. However, it can become very challenging too. An example is the huge work of cruise photographers, who take hundreds, if not thousands, of passengers photos during the day, and try to organize them during the night to sell them the day after to interested customers. A similar problem, with different time constraints, is addressed when organizing and keeping updated the usually huge photo collection of a magazine editorial office. When related operations can be performed in a totally automatic way, we speak of automatic face tagging. Of course, there is a tight relationship between automatic face tagging and face recognition. In general, face recognition requires comparing a sample image (or a template extracted from it) with a set of images (or templates) of known (enrolled) subjects to establish or verify the sample identity. Automatic face tagging is a particular application of face recognition: its aim is to identify the same person or group of persons in different images or video frames, with the purpose of organizing an archive accordingly, or of tracking the activities of a person. After an initial assignment of labels/tags performed by hand, the process of assigning an identity to a new incoming face and to classify the related image is performed automatically by the face recognition engine instead of a human operator. Clusters of images containing the same face are managed by the system, and new incoming images must be inserted in the correct cluster. Further applications for automatic face image tagging can also address video indexing and retrieval in large collections. Of course in all cases the accuracy of an automatic face image tagging system strongly depends on the face recognition algorithm used, and on its robustness to a variety of distortions (typically PIE – Pose, Illumination, Expression, but also ageing and other demographics). In particular, face tagging is a typical example of face recognition “in the wild” (Huang et al., 2007), due to the complete lack of control over the acquisition settings. Furthermore, tagging is typically performed at a time later the sample acquisition, so that in case of failure to recognize no kind of adjustment and re-acquisition is possible in any case, as for example in authentication applications. In this chapter we present some approaches to face recognition that seem better suited to address this problem. As we will see, automatic face tagging is a complex issue and this is the reason why this kind of systems have not reached a large distribution yet, and are not used in critical (security) settings. Some of the solutions to automatic face image tagging proposed are described in more detail, in order to give to the reader an overview of the state of the art.
Face recognition techniques may undergo different classifications. In Zhao et al. (2003) psychophysicists and neuroscientists theories are taken into account to identify three classes: holistic, which use the whole face region as input to the recognition system; feature-based methods, which first extract local features such as the eyes, nose, and mouth and then use their locations and local statistics for classification; hybrid, which work just as the human perception system is deemed to do, using both local features and the whole face region for recognition. A different classification was used in Abate et al. (2007), which mostly takes into account the main techniques underlying the literature approaches: linear/nonlinear projection methods (for dimensionality reduction), neural networks, Gabor filters and wavelets, fractals and Iterated Function Systems (IFSs), use of thermal and hyperspectral images. 3D face recognition is also often discussed, but it requires special equipment and/or computationally expensive processing, which is not appropriate for the kind of operations which we are addressing here. A detailed discussion of the methods in literature is out of the scope of this chapter, but the interested reader can find details in the cited works, and also in Nappi and Riccio (2008) and Jafri and Arabnia (2009).