Automatic Detection of Emotion in Music: Interaction with Emotionally Sensitive Machines

Automatic Detection of Emotion in Music: Interaction with Emotionally Sensitive Machines

Cyril Laurier (Universitat Pompeu Fabra, Spain) and Perfecto Herrera (Universitat Pompeu Fabra, Spain)
DOI: 10.4018/978-1-60566-354-8.ch002
OnDemand PDF Download:


Creating emotionally sensitive machines will significantly enhance the interaction between humans and machines. In this chapter we focus on enabling this ability for music. Music is extremely powerful to induce emotions. If machines can somehow apprehend emotions in music, it gives them a relevant competence to communicate with humans. In this chapter we review the theories of music and emotions. We detail different representations of musical emotions from the literature, together with related musical features. Then, we focus on techniques to detect the emotion in music from audio content. As a proof of concept, we detail a machine learning method to build such a system. We also review the current state of the art results, provide evaluations and give some insights into the possible applications and future trends of these techniques.
Chapter Preview

Section 1. Music And Emotions: Emotion In Music And Emotions From Music

To study the relationship between music and emotion, we have to consider the literature from many fields. Indeed, relevant scientific publications about this topic can be found in psychology, sociology, neuroscience, cognitive science, biology, musicology, machine learning and philosophy. We focus here on works aiming to understand the emotional process in music, and to represent and model the emotional space. We also detail the main results regarding the pertinent musical features and how they can be used to describe and convey emotions.

Key Terms in this Chapter

Music Categorization: models consider that perceptual, cognitive or emotional states associated with music listening can be defined by assigning them to one of many predefined categories. Categories are a basic survival tool, in order to reduce the complexity of the environment as they assign different physical states to the same class, and make possible the comparison between different states. It is by means of categories that musical ideas and objects are recognized, differentiated and understood. When applied to music and emotion, they imply that different emotional classes are identified and used to group pieces of music or excerpts according to them. Music categories are usually defined by means of present or absent musical features.

Musical Features: are the concepts, based on musical theory, music perception or signal processing, that are used to analyze, describe or transform a piece of music. Because of that, they constitute the building blocks of any Music Information Retrieval system. They can be global for a given piece of music (e.g., key or tonality), or can be time-varying (e.g., energy). Musical features have numerical or textual values associated. Their similarities and differences make possible to build predictive models of more complex or composite features, in a hierarchical way.

Supervised Learning: is a machine learning technique to automatically learn by example. A supervised learning algorithm generates a function predicting ouputs based on input observations. The function is generated from the training data. The training data is made of input observations and wanted outputs. Based on these examples the algorithm aims to generalize properly from the input/ouput observations to unobserved cases. We call it regression when the ouput is a continuous value and classification when the ouput is a label. Supervised learning is opposed to unsupervised learning, where the outputs are unknown. In that case, the algorithm aims to find structures in the data. There are many supervised learning algorithms such as Support Vector Machines, Nearest Neighbors, Decision trees, Naïve Bayes or Artificial Neural Network.

Music Information Retrieval: (MIR) is an interdisciplinary science aimed to studying the processes, systems and knowledge representations required for retrieving information from music. This music can be in symbolic format (e.g., a MIDI file), in audio format (e.g. an mp3 file), or in vector format (e.g., a scanned score). MIR research takes advantage of technologies and knowledge derived from signal processing, machine learning, music cognition, database management, human-computer interaction, music archiving or sociology of music.

Personal Music Assistants: are technical devices, that help its user to find relevant music, provide the right music at the right time and learn his profile and musical taste. Nowadays mp3 players are the music personal assistants, with eventually access to a recommendation engine. Adding new technologies like the ability to detect emotions, sense the mood and movements of the user will makes these devices “intelligent” and able to find music that triggers particular emotions.

Support Vector Machine: (SVM), is a supervised learning classification algorithm widely used in machine learning. It is known to be efficient, robust and to give relatively good performances. In the context of a two-class problem in n dimensions, the idea is to find the “best” hyperplane separating the points of the two classes. This hyperplane can be of n-1 dimensions and found in the feature space, in that case it is a linear classifier. Otherwise, it can be found in a transformed space of higher dimensionality using kernel methods. In that case we talk about a non-linear classifier. The position of new observations compared to the hyperplane tells us in which class is the new input.

Music Dimensional Models: consider that perceptual, cognitive or emotional states associated with music listening can be defined by a position in a continuous multidimensional space where each dimension stands for a fundamental property common to all the observed states. Pitch, for example, is considered to be defined by a height (how high or low in pitch it is a tone) and a chroma (the note class it belongs to, i.e., C, D, E, etc.) dimension. Two of the most accepted dimensions for describing emotions were proposed by Russel (Russel 1980): valence (positive versus negative affect) and arousal (low versus high level of activation). This variety of dimensions could be seen as the different expressions of a very small set of basic concepts.

Complete Chapter List

Search this Book:
Editorial Advisory Board
Table of Contents
Craig DeLancey
Jordi Vallverdú, David Casacuberta
Chapter 1
Oscar Deniz, Javier Lorenzo, Mario Hernández, Modesto Castrillón
Social intelligence seems to obviously require emotions. People have emotions, recognize them in others and also express them. A wealth of... Sample PDF
Emotional Modeling in an Interactive Robotic Head
Chapter 2
Cyril Laurier, Perfecto Herrera
Creating emotionally sensitive machines will significantly enhance the interaction between humans and machines. In this chapter we focus on enabling... Sample PDF
Automatic Detection of Emotion in Music: Interaction with Emotionally Sensitive Machines
Chapter 3
Christoph Bartneck, Michael J. Lyons
The human face plays a central role in most forms of natural human interaction so we may expect that computational methods for analysis of facial... Sample PDF
Facial Expression Analysis, Modeling and Synthesis: Overcoming the Limitations of Artificial Intelligence with the Art of the Soluble
Chapter 4
Sajal Chandra Banik, Keigo Watanabe, Maki K. Habib, Kiyotaka Izumi
Multi-robot team work is necessary for complex tasks which cannot be performed by a single robot. To get the required performance and reliability... Sample PDF
Multirobot Team Work with Benevolent Characters: The Roles of Emotions
Chapter 5
Matthias Scheutz, Paul Schermerhorn
Effective decision-making under real-world conditions can be very difficult as purely rational methods of decision-making are often not feasible or... Sample PDF
Affective Goal and Task Selection for Social Robots
Chapter 6
Christopher P. Lee-Johnson, Dale A. Carnegie
The hypothesis that artificial emotion-like mechanisms can improve the adaptive performance of robots and intelligent systems has gained... Sample PDF
Robotic Emotions: Navigation with Feeling
Chapter 7
C. Gros
All self-active living beings need to solve the motivational problem—the question of what to do at any moment of their life. For humans and... Sample PDF
Emotions, Diffusive Emotional Control and the Motivational Problem for Autonomous Cognitive Systems
Chapter 8
Bruce J. MacLennan
This chapter addresses the “Hard Problem” of consciousness in the context of robot emotions. The Hard Problem, as defined by Chalmers, refers to the... Sample PDF
Robots React, but Can They Feel?
Chapter 9
Mercedes García-Ordaz, Rocío Carrasco-Carrasco, Francisco José Martínez-López
It is contended here that the emotional elements and features of human reasoning should be taken into account when designing the personality of... Sample PDF
Personality and Emotions in Robotics from the Gender Perspective
Chapter 10
Antoni Gomila, Alberto Amengual
In this chapter we raise some of the moral issues involved in the current development of robotic autonomous agents. Starting from the connection... Sample PDF
Moral Emotions for Autonomous Agents
Chapter 11
Pietro Cipresso, Jean-Marie Dembele, Marco Villamira
In this work, we present an analytical model of hyper-inflated economies and develop a computational model that permits us to consider expectations... Sample PDF
An Emotional Perspective for Agent-Based Computational Economics
Chapter 12
Michel Aubé
The Commitment Theory of Emotions is issued from a careful scrutiny of emotional behavior in humans and animals, as reported in the literature on... Sample PDF
Unfolding Commitments Management: A Systemic View of Emotions
Chapter 13
Sigerist J. Rodríguez, Pilar Herrero, Olinto J. Rodríguez
Today, realism and coherence are highly searched qualities in agent’s behavior; but these qualities cannot be achieved completely without... Sample PDF
A Cognitive Appraisal Based Approach for Emotional Representation
Chapter 14
Clément Raïevsky, François Michaud
Emotion plays several important roles in the cognition of human beings and other life forms, and is therefore a legitimate inspiration for providing... Sample PDF
Emotion Generation Based on a Mismatch Theory of Emotions for Situated Agents
Chapter 15
Artificial Surprise  (pages 267-291)
Luis Macedo, Amilcar Cardoso, Rainer Reisenzein, Emiliano Lorini
This chapter reviews research on computational models of surprise. Part 1 begins with a description of the phenomenon of surprise in humans, reviews... Sample PDF
Artificial Surprise
Chapter 16
Tom Adi
A new theory of emotions is derived from the semantics of the language of emotions. The sound structures of 36 Old Arabic word roots that express... Sample PDF
A Theory of Emotions Based on Natural Language Semantics
Chapter 17
Huma Shah, Kevin Warwick
The Turing Test, originally configured as a game for a human to distinguish between an unseen and unheard man and woman, through a text-based... Sample PDF
Emotion in the Turing Test: A Downward Trend for Machines in Recent Loebner Prizes
Chapter 18
Félix Francisco Ramos Corchado, Héctor Rafael Orozco Aguirre, Luis Alfonso Razo Ruvalcaba
Emotions play an essential role in the cognitive processes of an avatar and are a crucial element for modeling its perception, learning, decision... Sample PDF
Artificial Emotional Intelligence in Virtual Creatures
Chapter 19
Sarantos I. Psycharis
In our study we collected data with respect to cognitive variables (learning outcome), metacognitive indicators (knowledge about cognition and... Sample PDF
Physics and Cognitive-Emotional-Metacognitive Variables: Learning Performance in the Environment of CTAT
Chapter 20
Anthony G. Francis Jr., Manish Mehta, Ashwin Ram
Believable agents designed for long-term interaction with human users need to adapt to them in a way which appears emotionally plausible while... Sample PDF
Emotional Memory and Adaptive Personalities
Chapter 21
Dorel Gorga, Daniel K. Schneider
The purpose of this contribution is to discuss conceptual issues and challenges related to the integration of emotional agents in the design of... Sample PDF
Computer-Based Learning Environments with Emotional Agents
Chapter 22
Emotional Ambient Media  (pages 443-459)
Artur Lugmayr, Tillmann Dorsch, Pabo Roman Humanes
The “medium is the message”: nowadays the medium as such is non-distinguishable from its presentation environment. However, what is the medium in an... Sample PDF
Emotional Ambient Media
Chapter 23
Jordi Vallverdú, David Casacuberta
During the previous stage of our research we developed a computer simulation (called ‘The Panic Room’ or, more simply, ‘TPR’) dealing with synthetic... Sample PDF
Modelling Hardwired Synthetic Emotions: TPR 2.0
Chapter 24
Cecile K.M. Crutzen, Hans-Werner Hein
A vision of future daily life is explored in Ambient Intelligence (AmI). It follows the assumption that information technology should disappear into... Sample PDF
Invisibility and Visibility: The Shadows of Artificial Intelligence
About the Contributors