3D Talking-Head Mobile App

Introduction

3D talking-head mobile app is basically referred to a mobile app that presents the head of a computer generated three dimensional animated character that can talk or hold a conversation with human users. The 3D talking-heads are head-like representation of three-dimensional models that can demonstrate the act of talking in computer generated realities, including virtual and augmented realities. The heads could either be realistic or stylized humanoid character which is featured through mobile app on smartphones, tablet computers or other mobile devices. The mobile app is commonly used for two purposes: language learning and entertainment (e.g. Figure 1). The app acts as a pedagogical agent or multimedia performer that replaces real human when interacting with human users.

Figure 1.

Teacher Johan is an example of 3D talking-head mobile app

To afford the interaction for learning or entertainment, a 3D talking-head mobile app consists of six components: animated 3D head model, voice over (VO) scripts, background audio, background graphics, navigational buttons, and instructional caption and subtitle (see Figure 2). The contents of the 3D talking-head mobile app are structured through the VO scripts, which are composed by subject matter experts or multimedia designers. Based on the learning or entertaining context depicted by the VO scripts, computer graphic artist designs a 3D head model and the background graphics for the mobile app; while sound designer composes background music and sound effects to match the intended context. The 3D head model is animated to demonstrate facial expressions and phoneme mouth shapes which can synchronize the VO scripts. In the creation of the mobile app, a computer programmer writes programming codes and scripts in authoring software to integrate the animated 3D head model, instructional caption and subtitle with the background audio and graphics. The programmer also develops a control scheme that configures the user interface and all navigational buttons for the mobile app. The integrated multimedia application will be published as a mobile app which is playable on specific operating systems (OS), such as Apple iOS, Android or Windows Phone OS.

Figure 2.

Components of a 3D talking-head mobile app

John Lewis (Lewis, 1991) at Computer Graphics Laboratory, the New York Institute of Technology and Georg Trogemann (Frank, Hoch & Trogemann, 1997) in the Academy of Media Arts Cologne, Germany are among the earliest examining topics related to 3D talking heads; while a research team led by Ahmad Zamzuri Mohamad Ali (Ahmad Zamzuri Mohamad Ali & Segaran, 2013) at Sultan Idris Education University, Malaysia and Riccardo Scateni (Sorrentino & Scateni, 2012) at University of Cagliari, Italy are among the leading experts in the area of 3D talking-head mobile app.

Overview

The potential of 3D talking-head began to capture academics’ attention in the 1990s. Most of the research studies were focusing on two aspects—lip synchronization and facial expression of 3D models, which would determine the authenticity and the accuracy of 3D talking-head mobile app. Since mobile apps were generally non-exist in the 1990s, the 3D talking-heads were either web-based or limited to personal computers, i.e. desktop computers and laptops.