Eye-Gaze and Facial Expressions as Feedback Signals in Educational Interactions

Introduction

With the development of personal computer technology in the 1980’s, also educational applications started to get developed. Personal computers allowed educators to design and develop their own software, and various applications emerged for both instructional and support purposes; see an overview of the history and design of the educational systems in Reiser (2001a, b). Since the early applications, however, technology has taken big leaps, and especially interface technology has advanced much beyond text and mouse input. For instance, the techniques regarding the perception of the user, such as eye-gaze and face tracking, have become robust and easier to use in ordinary applications. They have been used in game applications (Isokoski et al., 2009), although their use in educational applications is not so common (however, some related research is conducted with respect to translation, see e.g. Hyrskykari et al., 2000).

In this chapter, we explore possibilities to use novel face and gaze tracking technology in educational applications. Our goal is to contribute to education technology by integrating novel interface technology and dialogue modelling techniques into a practical interactive learning tool that assists students in second language learning (L2-learning). Language learning differs from math and physics learning in that the main goal is to learn to communicate with other people using the new language, i.e. to learn to use the new language as a tool in everyday social situations. From this perspective, the support device for a language learner must be highly interactive, something that allows learning and training in interactive settings, and also learning of the interaction itself. Here we approach Jonassen (2000) who, in his review article, searchers for a new transformative view of educational technology. Jonassen advocates the constructivist view of learning, according to which teaching is not just transmitting knowledge from the teacher to the student, but a two-way dynamic situation where the knowledge is constructed through interaction and communication. Jonassen mainly focuses on collaborative learning, but it is clear that in L2-learning, dynamic interaction with the teacher is crucial, since it essentially sets out the practices and strategies which the student should adopt as part of the communicative repertoire for the language to be learnt.

Communication is social activity, bound by social and cultural norms and obligations, and learning, in particular L2-learning, is also governed by communicative practices. Because of this, it is important to support L2 learners’ pragmatic and interactional competence, and to develop their awareness of the differences between native and target languages both on lexical-syntactic and pragmatic levels. The learning of various patterns and practices of language use, including structure and organization of interactions, requires pedagogy that Hall (1999) has described as “prosaics of interaction”: the learners’ attention is directed to the pertinent features in the interaction context and linguistic resources, and through the systematic study of the interactive practices, the learners are thus helped to detect relevant patterns. Hall proposes that appropriate pedagogical activities in this respect should include analysis of videotapes, following the methodology of linguistic anthropology and discourse analysis. Furthermore, the analysis should concern recurring and goal-directed interactions like mealtime talk, faculty meetings, advice sessions, service encounters, etc., since the students, at least in the beginning of their studies, may not be able to detect the necessary features in less ritualized language uses. According to Hall (1999), the situations function as “cultural maps” of the social environment, and introduce the language learner, a newcomer, to relevant socio-cultural information.

While it is vital for students to develop a critical awareness of the language use, it is also important that they can take part in different communicative situations themselves, and become familiar with the appropriate use of linguistic resources in real interactions. Often suitable human partners may not be available for this kind of practise purposes, especially for beginners, who mostly need drill exercises to master simple interactions rather than complicated exchanges of information. For such repetitive learning situations, automated agents can prove useful assistants, as they can provide practise anytime without getting tired or frustrated.