Modeling and Synthesis of Realistic Visual Speech in 3D

Modeling and Synthesis of Realistic Visual Speech in 3D

Gregor A. Kalberer (BIWI-Computer Vision Lab, Switzerland), Pascal Müller (BIWI – Computer Vision Lab, Switzerland) and Luc Van Gool (BIWI – Computer Vision Lab, Switzerland)
DOI: 10.4018/978-1-59140-299-2.ch008
OnDemand PDF Download:


The problem of realistic face animation is a difficult one. This is hampering a further breakthrough of some high-tech domains, such as special effects in the movies, the use of 3D face models in communications, the use of avatars and likenesses in virtual reality, and the production of games with more subtle scenarios. This work attempts to improve on the current state-of-the-art in face animation, especially for the creation of highly realistic lip and speech-related motions. To that end, 3D models of faces are used and — based on the latest technology — speech-related 3D face motion will be learned from examples. Thus, the chapter subscribes to the surging field of image-based modeling and widens its scope to include animation. The exploitation of detailed 3D motion sequences is quite unique, thereby narrowing the gap between modeling and animation. From measured 3D face deformations around the mouth area, typical motions are extracted for different “visemes”. Visemes are the basic motion patterns observed for speech and are comparable to the phonemes of auditory speech. The visemes are studied with sufficient detail to also cover natural variations and differences between individuals. Furthermore, the transition between visemes is analyzed in terms of co-articulation effects, i.e., the visual blending of visemes as required for fluent, natural speech. The work presented in this chapter also encompasses the animation of faces for which no visemes have been observed and extracted. The “transplantation” of visemes to novel faces for which no viseme data have been recorded and for which only a static 3D model is available allows for the animation of faces without an extensive learning procedure for each individual.

Complete Chapter List

Search this Book:
Table of Contents
Chapter 1
Angel Sappa, Sotiris Malassiotis
This chapter presents a survey of the most recent vision-based human body modeling techniques. It includes sections covering the topics of 3D human... Sample PDF
Advances in Vision-Based Human Body Modeling
Chapter 2
Marius Preda, Ioan A. Salomie, Françoise Preteux, Gauthier Lafruit
Besides being one of the well-known audio/video coding techniques, MPEG-4 provides additional coding tools dedicated to virtual character animation.... Sample PDF
Virtual Character Definition and Animation within the MPEG-4 Standard
Chapter 3
B. J. Lei, E. A. Hendriks, Aggelos K. Katsaggelos
This chapter presents an extensive overview of passive camera calibration techniques. Starting with a detailed introduction and mathematical... Sample PDF
Camera Calibration for 3D Reconstruction and View Transformation
Chapter 4
Burak Ozer, Tiehan Lv, Wayne Wolf
This chapter focuses on real-time processing techniques for the reconstruction of visual information from multiple views and its analysis for human... Sample PDF
Real-Time Analysis of Human Body Parts and Gesture-Activity Recognition in 3D
Chapter 5
Kostas Karpouzis, Amaryllis Raouzaiou, Athanasios Drosopoulos, Spiros Ioannou, Themis Balomenos, Nicolas Tsapatsoulis
This chapter presents a holistic approach to emotion modeling and analysis and their applications in Man-Machine Interaction applications. Beginning... Sample PDF
Facial Expression and Gesture Analysis for Emotionally-Rich Man-Machine Interaction
Chapter 6
Ana C. Andres del Valle, Jean-Luc Dugelay
This chapter presents a state-of-the-art compilation on facial motion and expression analysis. The core of the chapter includes the description and... Sample PDF
Techniques for Face Motion & Expression Analysis on Monocular Images
Chapter 7
Peter Eisert
In this chapter, the state-of-the-art in facial animation and expression analysis is reviewed and new techniques for the estimation of 3-D human... Sample PDF
Analysis and Synthesis of Facial Expressions
Chapter 8
Gregor A. Kalberer, Pascal Müller, Luc Van Gool
The problem of realistic face animation is a difficult one. This is hampering a further breakthrough of some high-tech domains, such as special... Sample PDF
Modeling and Synthesis of Realistic Visual Speech in 3D
Chapter 9
Markus Kampmann, Liang Zhang
This chapter introduces a complete framework for automatic adaptation of a 3D face model to a human face for visual communication applications like... Sample PDF
Automatic 3D Face Model Adaptation with Two Complexity Modes for Visual Communication
Chapter 10
Zhen Wen, Pengyu Hong, Jilin Tu, Thomas S. Huang
This chapter presents a unified framework for machine-learning-based facial deformation modeling, analysis and synthesis. It enables flexible... Sample PDF
Learning 3D Face Deformation Model: Methods and Applications
Chapter 11
Nikos Karatzoulis, Costas T. Davarakis, Dimitrios Tzovaras
This chapter presents a number of promising applications and provides an overview of recent developments and techniques in the area of analysis and... Sample PDF
Synthesis and Analysis Techniques for the Human Body: R&D Projects
About the Authors