Recognizing Student Emotions using Brainwaves and Mouse Behavior Data

Introduction

Students experience various emotions while engaged in learning. Such emotions, also referred to as academic emotions (Pekrun, 2002), may affect the flow of learning as well as the motivation to continue with the learning task. This has been the challenge for the human tutors and even for those who develop intelligent tutoring systems (ITS). Indeed, effective tutors, whether human tutors or computer-based intelligent tutoring systems, are those who are not only aware of the cognitive needs of the students but also of their affective needs.

With this in mind, recent research projects in the area of ITS have tried to address not only the cognitive needs of students but their affective needs as well. Such affective systems, also referred to as affective tutoring systems, consider the effect of emotions in the learning process of a learner as well as the typical emotional patterns under different learning scenarios. Academic emotions typically experienced by a learner while using a tutoring system are confidence (Arroyo et al., 2009; Azcarraga et al., 2011a, 2011b, 2011c; Ibañez et al., 2011), excitement (Arroyo et al., 2009; Azcarraga et al., 2011a, 2011b, 2011c; Ibañez et al., 2011), frustration (Arroyo et al., 2009; Azcarraga et al., 2011a, 2011b, 2011c; Burleson, 2006; Ibañez et al., 2011), interest (Arroyo et al., 2009; Azcarraga et al., 2011a, 2011b, 2011c; D’Mello & Graesser, 2009; Ibañez et al., 2011; Kapoor, Burleson & Picard, 2007) , flow/engagement (D’Mello & Graesser, 2009; Stevens, Galloway & Berka, 2007), boredom (D’Mello & Graesser, 2009) and confusion (D’Mello & Graesser, 2009).

Affective tutoring systems are capable of recognizing student affect based on tutorial information complemented with the user profile. Furthermore, these systems sometimes also include a combination of facial expression, gesture and physiological signals. In (Arroyo et al., 2009), affective states such as confident, frustrated, excited and interested are predicted with high accuracy using special devices such as a camera to capture facial expression, posture chair to monitor the level of engagement, pressure-sensitive mouse and skin-conductance sensor. Similarly, Burleson (2006) uses the same set of devices in order to predict student frustration and the need for help. Moreover, tutorial information such as conversational cues, posture and facial features are used in Autotutor to predict boredom, flow/engagement, confusion and frustration (D’Mello & Graesser, 2009).