Music Control in an Interactive Conducting System Using Kinect

Music Control in an Interactive Conducting System Using Kinect

Yi-Shin Chen (Institute of Information Systems and Applications, National Tsing Hua University, Hsinchu, Taiwan), Leng-Wee Toh (Department of Computer Science, National Tsing Hua University, Hsinchu, Taiwan) and Yi-Lan Liu (Institute of Information Systems and Applications, National Tsing Hua University, Hsinchu, Taiwan)
DOI: 10.4018/ijmdem.2013100103

Abstract

Music conducting is the art of directing musical ensembles with hand gestures to personalize and diversify a piece of music. Although the ability to successfully perform a musical piece demands intense training and coordination for the conductor and the orchestra, preparing a practice session is expensive and time-consuming. Hence, there is a genuine need for alternatives capable of providing adequate training for conductors at all skill levels. The potential use of virtual and augmented reality technology holds particular promise. The goal of this research is to examine the mechanics of music conducting and to develop a system capable of closely simulating the experience of conducting a piece of music. After extensive discussions with professional and nonprofessional conductors, in addition to wide-ranging research regarding music conducting materials, several key features of conducting were identified. A set of lightweight algorithms exploring these features was developed to enable tempo control, volume adjustment, and instrument emphasis, which are core components of conducting. Such a system would be a helpful training tool for students, an experiential tool allowing professional conductors and composers to shape music at a low cost, or an entertainment tool for nonprofessional music lovers. In this paper, we propose a real-time interactive conducting system using Microsoft Kinect. The proposed system overcomes the limitation of Kinect's design, which is generally designed for large body movements. In this system, delicate conducting signals can be correctly recognized without referencing any prior knowledge. Evaluation of the algorithms in real-world scenarios reveals promising results. The system was evaluated by conductors of all skill levels and provided a high level of accuracy and a low latency. Users of the final system expressed satisfaction with the virtual experience.
Article Preview

Introduction

The increasing significance of cooperation and communication in the international world of music is a significant trend. The barrier of geographical distances and the cost and efficiency of cooperation and communication processes are pressing global issues. Numerous advances in multimedia and human computer interaction (HCI) technology have introduced a number of innovative benefits to our lives. Remote meeting tools and interactive e-learning platforms are good examples of cost-effective applications that are now used around the world to benefit business and education areas.

To extend online collaborative opportunities to music professions, a few software applications have been developed that use distributed environments, which allow real-time interactions between geographically distant musicians (Chafe, Wilson, Leistikow, Chisholm, & Scavone, 2000; Renaud, Carot, & Rebelo, 2007; Zimmermann, Chew, Arslan, & Moses, 2008). However, synchronization challenges constrain the number of players in different locations and make distributed orchestra rehearsal impractical. To mitigate synchronization challenges, an interactive conducting system could become the middle agent in interpreting conductor gestures and tuning music based on recognized results for each distributed player. In addition to being a cheap and practical solution for amateur conductors, a music conducting system could be a helpful teaching and training tool in schools and academic environments. Even professional conductors could use a conducting system with musicians' audio files as input to experiment with the shape of the music. Conducting systems could further enhance composers' ability to experience their own creations.

A number of conducting systems were completed in recent decades that assist in the construction of interactive conducting systems. While some prior efforts emphasized understanding the emotional expressions conveyed through conducting techniques, others focused on building the actual systems. The Conductor's Jacket (Nakra, 2000) is a relatively successful system available for use that provides a test bed for the study of emotional expression in music conducting. After collecting and analyzing data from conductors during rehearsals and performances, thirty-five significant features that reflected intuitive and natural gestural tendencies were identified. These features then assist the conducting systems in better interpreting conducting gestures.

Several works have been published on constructing interactive conducting systems (Argueta, Ko &, Chen, 2009; Borchers, Lee, Samminger, & Müuhlhäuser, 2004; Lee, Grüll, Kiel, & Borchers, 2006a; Lee et al., 2006b; Murphy, Andersen, & Jensen, 2003; Nakra, Ivanov, Smaragdis, & Ault, 2009; Prisco, Sabatino, Zaccagnino, & Zaccagnino, 2011; Segen, Gluckman, & Kumar, 2000; Wong, Yuen, & Choy, 2008). In the last decade, the construction of interactive conducting systems has facilitated conductor-specialized hardware. A good interactive conducting system should provide tempo control, volume control, instrument emphasis, and pattern-changing beat recognition. Moreover, it should be suitable for both beginners and professionals.

Complete Article List

Search this Journal:
Reset
Open Access Articles: Forthcoming
Volume 10: 4 Issues (2019): Forthcoming, Available for Pre-Order
Volume 9: 4 Issues (2018): 2 Released, 2 Forthcoming
Volume 8: 4 Issues (2017)
Volume 7: 4 Issues (2016)
Volume 6: 4 Issues (2015)
Volume 5: 4 Issues (2014)
Volume 4: 4 Issues (2013)
Volume 3: 4 Issues (2012)
Volume 2: 4 Issues (2011)
Volume 1: 4 Issues (2010)
View Complete Journal Contents Listing