Organ Augmented Reality: Audio-Graphical Augmentation of a Classical Instrument

Introduction

Augmented musical instruments are traditional instruments that are modified by adding controls and additional outputs such as animated graphics (Bouillot et al., 2009; Thompson et al., 2007). The problem with usual approaches to instrument augmentation is that it generally makes the instrument more complex to play and more complex to understand by the spectators. The enhanced functionality of the instrument often distorts the perceived link between the performer’s actions and the resulting sounds and images. Augmentation is likely to confuse the audience because it lacks transparency and legibility.

In addition to augmenting traditional instruments with new controllers, like the hyper-kalimba (Rocha et al., 2009) which extends the kalimba (an instrument from the percussion family), Augmented Reality is also used to create new musical instruments. Some of these instruments mimic real music devices like the Digital Baton (Marrin et al., 1997), replicating the traditional conducting baton, or the AR scratching¹ imitating a DJ’s vinyl scratch. Other musical instruments that use Augmented Reality are totally innovative and are not based on existing devices. The Augmented Groove (Poupyrev et al., 2001) is an example of such a device where novice users manipulate a physical object in space to play electronic musical compositions. The main difference between creating novel instruments and extending existing instruments is the level of familiarity with the instrument. Instrument extension seems more suitable for experimented performers rather than novice ones due to the experience level with the instrument and possibly a wider range of control.

Musical instrument augmentation is interesting because it extends a traditional instrument, while preserving and enriching its performance and composition practices. The Organ and Augmented Reality (ORA) project focuses on a rarely stressed use of augmentation, the enhanced comprehension and legibility of a music instrument without increasing its complexity and opacity. Our research on output augmentation follows the same purposes as (Jordà, 2003), making the complexity of music more accessible to a larger public. Jorda’s work focused on the playing experience; similarly, we intend to improve and facilitate the listening experience. These principles have been used by Jordà et al. (2007) for the design of the ReacTable, an augmented input controller for electronic musical instruments. The ReacTable is a legible, graspable, and tangible control interface, which facilitates the use of an electronic instrument so as to be accessible to novices. Its use by professionals in live performances confirms that transparency is not boring and is compatible with long term use of the instrument.

This paper presents the issues and technical details of the ORA project and performance, the augmentation of an historical church organ for a better understanding and perception of the instrument through intuitive visual and audio outputs. It is based on the following achievements:

• •

  The visuals are directly projected onto the organ pipes (not on peripheral screens),

• •

  The visual augmentation is temporally and spatially aligned: the visual rendering is cross-modally synchronized with the acoustic signal and the graphical projection is accurately aligned with the organ geometry,

• •

  The augmentation preserves the traditional organ play. Traditional compositions as well as new artworks can be played on the augmented instrument,

• •

  The augmentation offers a better understanding of the instrument’s principles by showing a visualization of hidden data such as the spectral content of the sound and its position inside the instrument.