Developing Educative Multimodal Conversational Applications for Mobile Devices

Introduction

Technology integration is an increasingly crucial element of teaching and learning. Devices such as smartphones, tablets and wearables are becoming widespread, and the fact that increasingly more individuals have always with them a device with numerous displays, sensors and connectivity possibilities, opens new learning scenarios that demand more sophisticated interfaces (Pérez-Marín & Pascual-Nieto, 2011; Oulasvirta et al., 2012; Becker et al., 2013).

Applications aimed at introducing these technologies and devices to improve teaching and learning have mostly opted for the use of multimodal interfaces (O’Halloran, 2015; Salse et al., 2015). Multimodal conversational agents (McTear et al., 2016; Griol et al., 2014; López-Cózar & Araki, 2005) can be defined as computer programs designed to emulate communication capabilities of a human being including natural language and several communication modalities (such as speech and visual information).

These interfaces employ a variety of techniques to engage students in learning. Using natural language in educational software also allows students to spend their cognitive resources on the learning task, rather than in how to use the interface of the application. Also, as a complement to teacher-student communication in online scenarios and out of school studying hours, dialog and anthropomorphic characteristics of pedagogical multimodal systems may help supporting the social dimension of the learning activities, and the social context has been argued to help the cultivation of, and motivation for, knowledge (Fryer & Carpenter, 2006; Baylor & Kim, 2005). In addition, the combination of conversational interfaces and mobile devices has lead to a new paradigm in which they can collect information from the user pervasively. This can help to build more complex user and dialog models to be employed not only to provide the system functionality, but also to boost its performance.

Multimodal conversational systems developed to provide these functionalities typically rely on a variety of components, such as speech recognition and synthesis engines, natural language processing components, dialog management, databases management, and graphical user interfaces. Laboratory systems usually include specific modules of the research teams that build them, which make portability difficult. Thus, it is a challenge to package up these components so that novice users with limited engineering resources can easily install them.

In this chapter, we describe a framework to develop pedagogic conversational interfaces for mobile devices. It provides a modular and scalable architecture to develop such systems efficiently for Android-based mobile devices. The proposed framework integrates the facilities of the Android API in a modular architecture that emphasizes interaction management and context-awareness to build sophisticated, robust and maintainable educative applications. Our proposal for context-awareness is based on the definition of a statistical methodology for user modeling that estimates the user intention during the dialog. In our contribution, the information provided by the user model can also enriched by considering information related to the external context of the interaction.

We will also describe the practical application of the proposed framework to develop a mobile learning application that provides a range of functionalities covering the provision of personalized information, access to social networks, resolution of different kinds of educative exercises, etc. The application is focused on the generation of practical educative exercises that promote autonomous learning and self-assessment. Our proposal to develop pedagogic conversational interfaces improves the interaction to solve the proposed exercises, personalize the selection of the different learning activities considering the student’s preferences using the system and their previous uses, and allows the provision of immediate feedback by means of the automatic assessment of the learning activities.