Emotional State Recognition Using Facial Expression, Voice, and Physiological Signal

Emotion Recognition System Architecture

The analysis of existing emotion recognition systems reveals a decomposition into three levels, each fulfilling a specific function: Capture, analysis and interpretation levels. Figure 1 shows the emotion recognition system architecture.

Figure 1.

Emotion recognition system architecture

At the capture level, the information is captured from the real world and in particular from the user through devices (camera, microphone, etc.). This information is then analyzed in the analysis level, where emotionally relevant characteristics are extracted from the captured data. Finally, the extracted characteristics are interpreted to obtain an emotion. This division into three level - capture, analysis and interpretation - is classic in emotions recognition and form a functional motif on which we rely to develop a model.

This architecture model offers five component types (Figure 2). Each component subscribes and issue one or more data stream. The capture unit has the role of interfacing with a physical device for capturing data. The feature extractor analyzes input data in order to extract one or more emotionally relevant characteristics. An interpreter receives the values of several characteristics. Its role is to interpret emotion. This interpretation is subject to the emotion model considered (discrete model, continuous, or componential) as well as the computer algorithm used (e.g., neural network, hidden Markov model, etc.).

The model also has two types of components unrelated to an “emotion recognition” logic. The role of the adapter is to modify a data flow. It can be a simple modification of format as a heavy processing unrelated to the recognition (3D tracking by camera for example). The purpose of the hub is to amalgamate several data stream according to an ad hoc strategy.

Figure 2.

The five components of recognition architecture