Game Sound Technology and Player Interaction: Concepts and Developments

Game Sound Technology and Player Interaction: Concepts and Developments

Mark Grimshaw (University of Bolton, UK)
Release Date: September, 2010|Copyright: © 2011 |Pages: 504
ISBN13: 9781616928285|ISBN10: 161692828X|EISBN13: 9781616928308|DOI: 10.4018/978-1-61692-828-5


The sounds that a player hears while immersed in a game can have dramatic effects on that person, opening up a considerable, far-reaching field of research.

Game Sound Technology and Player Interaction: Concepts and Developments researches both how game sound affects a player psychologically, emotionally, and physiologically, and how this relationship itself impacts the design of computer game sound and the development of technology. This compilation also applies beyond the realm of video games to other types of immersive sound, such as soundscape design, gambling machines, emotive and fantastical sound to name a few. The application for this research is wide-ranging, interdisciplinary, and of primary importance for academics and practitioners searching for the right sounds.

Topics Covered

The many academic areas covered in this publication include, but are not limited to:

  • Emotion, content, context for sound
  • Gambling machine sound
  • Natural to synthetic soundscape
  • Perceived quality
  • Procedural audio
  • Sound and fear
  • Sound design
  • Sound for fantasy
  • Sound in games
  • Soundscape design

Table of Contents and List of Contributors

Search this Book:


The human ability to become immersed in fantasy is often described as “the willing suspension of disbelief,” a phrase Samuel Taylor Coleridge first coined in the early 19th Century as an argument for the fantastical in prosody and poetry.  What is a computer game?  At base, it is nothing more than a cheap plastic disc encased within a cheaper plastic tray.  And the system it is destined for?  A box of electronics, lifeless in a corner.  Put the two together, though, throw in the player's imagination and interaction, and he or she is delivered of experiences that, to use Diderot's phrase, are “the strongest magic of art.”  Disbelief is suspended willingly, sense and rationality recede, and the player becomes engaged with, engrossed in, and, given the appropriate game, immersed in a virtual world of flickering light and alluring sound where the fantastical becomes the norm and the mythic reality.

For the reader interested in that flickering light, there is a plethora of books and scholarly articles on the subject.  (It is a curious fact that, until recently, most academic books on computer games treated the medium as spectacle: one would think the authors deaf, the sparse attention paid to sound.)  For the reader interested in the ins and outs of music and sound software, how to rig a microphone to record sound, and how to transfer that sound to a game environment, there likewise is a wealth of handy resources.  For the reader truly interested in understanding or harnessing the power of sound in that virtual world, in emulating reality or the creation of other realities, in engaging, engrossing and immersing the player through sound and emotion, there is this book.  As Joseph Conrad states:  “The power of sound has always been greater than the power of sense” (A Personal Record, 1912).

This book deals with computer game sound by focusing on sound FX rather than game music, except where the music is interactive or otherwise intimately bound up with the playing of the game.  Such sound FX may be used to emulate acoustic environments of the real world while others are used to create alternate realities, some are based upon the use of audio samples whilst others are starting to make use of procedural synthesis and audio processing, some sound works hand-in-hand with image and game action to immerse the game player in the gameworld while, in other cases, the production of interactive music is the sole raison d’être of the game.  From the simplest of puzzle games to the most detailed and convoluted of gameworlds, sound is the indicator par excellence of player engagement and interaction with the structures of the game and the rules of play.

Academic writing about game sound, especially its analytical and theoretical underpinnings, is a developing area and this is reflected by the diversity of theoretical methodologies and the variety of terminology in use.  Far from being a weakness, this range points to the potential for the discipline and the broad appeal of its study because it is, at heart, multidisciplinary.  The range of subject matter across the chapters reflects the complexity and potential of human interaction with sound in virtual worlds as much as it reflects the passions, backgrounds, and training of the book's contributors.  Their contributions to the study of computer game sound bring in approaches and theories from film studies, cultural studies, sound design, acoustic ecology, acoustics, systems design and computer programming, and cognitive sciences and psychology.  The authors themselves have a diversity of experience: some are researchers and academics whilst others are sound practitioners in the games industry.  All are experts in their chosen field, yet all are students of game sound, forever exploring, forever questioning, forever seeking to drive the study and practice forward.

The readers of this book may be similarly diverse in terms of both discipline and motivation.  There is something for everyone here:  the university student for whom a knowledge of computer game sound leads to that important qualification, a game sound designer wishing to keep abreast of the latest thinking and developmental concepts, or an academic theoretician or researcher working to innovate game sound theory or technology.  Furthermore, the appeal of the book is wider than computer games, reaching out to those working in virtual reality or with autism, for example.  Readers will not find screeds of instructions for software or hardware, programming recipes or tips on how to break into the industry.  Instead, they will find lucid essays on philosophical questions, theoretical analyses on aspects of computer game sound, models for conceptualizing sound, ideas for sound design, and provocative discussions about new sound technology and its future implications.  All chapters raise further questions about the fascinating relationship between player and sound.

Reflecting the disciplines the authors come from, some key terms (found at the back of each chapter) are provided with definitions that, prima facie, differ slightly from the definition provided for the same key term in another chapter.  As with the authors' preferences for American or British English, this has been allowed to stand in order to illustrate both the diversity of approaches to the topic throughout the book and the educational and professional backgrounds of each author.  The study of computer game sound is yet young and the terminology and its application still in flux: in this book the definition for each key term is tailored to the chapter it belongs to.

The term “computer game” has been chosen, in preference to a number of other possibilities, as referring to all forms of digital game, arcade machine, gaming console, PC game, or videogame and the reader may assume that, unless a chapter uses one of those specific forms, “computer game” references the general case.  Quite deliberately, the term has been chosen in preference to videogame in order to fly the flag for sound: videogames are not just video but sound too, and all chapters proselytize the importance of sound to the game experience even where they reference the relationship of sound to image.  “Sound” has generally been chosen in preference to “audio” because the focus of the book is on the relationship between sound and player rather than techniques for creating and manipulating audio data.  However, “audio” is the usual term in some disciplines and, here, authors have been given free reign to use whichever terminology they are comfortable with.

The book itself is organized into five sections.  None is mutually exclusive in terms of its content.  Indeed, the astute reader will pick up divers common threads meandering their way through the chapters: the debt game sound owes to film sound and the need to slough off that used skin; issues of presence and player immersion; realism; the unique, interactive nature of computer game sound; and the potential for the emotional manipulation of the player, for instance.  All chapters, too, have an eye on the future, and authors have been encouraged to speculate on its possibilities.

An oft-overlooked area in computer gaming (and certainly not the first thing that comes to mind with the term “computer game”) is that of electronic gambling machines: one-armed bandits and their modern equivalents. Karen Collins and her co-authors open the first section on Interactive Practice by providing a fascinating glimpse into how music and sound FX is used to provoke and toy with the user's emotions in an effort to part them from their money. They draw parallels to sound use in other, more typical computer games and suggest ways in which sound use in electronic gambling machines might provide inspiration for the design and analysis of sound in computer games in general.

Mats Liljedahl's chapter is an attempt to redress the imbalance between visual and auditory modes in computer games.  It does this by providing an overview of the use of sound in virtual environments and augmented realities, in particular, concentrating on the sound designer's required attention to emotion and flow.  Using the concept of GameFlow, Liljedahl describes and explains two games he helped design in which the sound modality is purposefully given priority over the visual.  The chapter seeks to inspire and serves as an introduction to the art of computer game sound design: Sound for Fantasy and Freedom.

Linda O Keeffe takes a holistic view of computer game sound by treating it as a dynamic soundscape created anew at each playing.  She draws upon soundscape and acoustic ecology theory to elucidate her stance and compares and contrasts game soundscapes to real-world soundscapes.  Throughout, O Keeffe prompts questions regarding the listener's perception of, and relationship to, soundscapes: what is noise, what roles do context and the player's culture and society play?  Ultimately, it asks how can we (and why should we) design immersive soundscapes for the gameworld.

Next, Axel Berndt takes a close look at diegetic music in games, design principles for music games and issues of interactivity and algorithmic performance.  The chapter presents a critique on the performance of in-game music in recent and current games and offers advice and solutions to improve what is currently a rather static state of affairs, merely scratching at the surface of possibility. Interactivity in music games is assessed through a critique of what is termed visualized music, and Berndt proposes a framework of design that incorporates musical performance paradigms both as individual and as social, collaborative practice.

The second section, Frameworks & Models, begins with a chapter by Kristine Jørgensen, which is both an exhaustive survey of the use of diegetic terminology, with regard to game sound, and a proposal for a new conceptual model for such sound.  The main thrust of her argument is that the concepts of diegetic sound and non-diegetic sound have been transposed from film theory to the study of computer games with frequently scant regard for the very different premises of the two media.  Games are not films and the use-value of game sound is greater than that of film sound. The interactive, real-time nature of computer games and the immersive environments of many game genres requires a radical reappraisal of sound usage and sound design for games.

Ulf Wilhelmsson and Jacob Wallén follow with a chapter that proposes a model for the analysis and design of computer game sound which combines two previous models—the IEZA Framework for game sound and Walter Murch's conceptual model for film sound—with affordance and cognition theories.  The IEZA Framework accounts for the structural basis of game sound, the function of sound, while Murch's model describes sound as either embodied or encoded, a system that accounts for human perception and cognitive load limits.  Combining the two systems, the authors assert, provides a powerful tool not only for analysis but also for the planning and design of computer game sound, and this claim is demonstrated by a practical example.

In her chapter, Milena Droumeva filters the computer game soundscape through precepts of Schafer's and Truax's soundscape and acoustic communication theories. Different approaches to listening to game sound are evaluated for their role in the perception of realism: Does sound provide fidelity to source or does it provide a sense of verisimilitude? What are the strengths of each approach as regards computer game sound design?  Droumeva ultimately advocates a view comprising game soundscape and player together as an acoustic ecology and then expands that ecology from the virtual world of the game to include concurrent sounds from the real world.

Ulrich Reiter's chapter on Perceived Quality in Game Audio explores the bi-modality and cross-modality of auditory and visual stimuli in gameworlds. It summarizes previous work in this area and proposes a high-level salience model for the design of audio in games that accounts for both interactivity and attention as the bases for evaluating audio quality.  Issues of level of realism and verisimilitude are discussed while the validity and uses of Reiter's proposed model are substantiated through experimental methods outlined towards the end of the chapter.

Paul Toprac and Ahmed Abdel-Meguid begin the section dealing with Emotion & Affect.  The authors present four relevant emotion theories and summarize fundamental research they conducted in order to test properties of diegetic sound best suited to evoke sensations of fear, anxiety, and suspense. Their results—an early example of an empirical and statistical basis for sonic fear and anxiety—lead the authors to devise some rough heuristics for the design of such emotions into computer game sound and to point to directions for future research in the area.

The next chapter, by Guillaume Roux-Girard, also deals with the perception of fear in computer game sound. It provides an in-depth analysis of sound usage in the survival horror game genre, focusing on sound's ability to instil fear and dread in the player. It proposes a user-centric model for sound based upon film sound practice and existing models for computer game sound, one based on the reception of sound rather than its production. Roux-Girard provides several illustrative examples from recent horror games to validate the model.

In Uncanny Speech, Angela Tinwell, Mark Grimshaw, and Andrew Williams continue exploring horror with a look at Non-Player Character speech in horror games and its relationship to the 1970s' theory of the Uncanny Valley.  The authors trace the development of theories of the uncanny from their beginnings in psychoanalysis over 100 years ago through to the practical application of Uncanny Valley theory in robotics and explore its strong correlation to fear and anxiety in computer games. Recent empirical work by the authors is described, and its implication for the design and production of Non-Player Character speech in computer games is discussed.

Stuart Cunningham, Vic Grout, and Richard Picking's chapter, Emotion, Content, and Context in Sound and Music, is an exploration of the interaction possible between player and computer game sound, especially music playlists used in conjunction with games.  The authors provide an overview of emotion research in the context of computer games and consider the emotional and affective value of sound and music to the player.  The experimental work summarized in the chapter includes generating musical playlists according to the environmental context of the player outside the game. Not only does this allow intriguing sensory and perceptual overlap and interplay between real-world and virtual, but the authors also suggest further possibilities such as the playlists themselves  responding to the changing psychology and physiology of the player during gameplay.

Concluding the section, Lennart Nacke and Mark Grimshaw's chapter is a study of computer games as affective activity.  The chapter focuses on how sound affects, indeed effects flow and, particularly, immersion and supplies a preliminary mathematical equation for modelling the latter.  The authors start with a review of psychological, physiological, and psychophysiological experiments on the effects of sound and image in virtual environments.  Following a summary of a recent psychophysiological study on computer game sound conducted by the authors, the chapter concludes with a discussion about the advantages and disadvantages of such an empirical methodology before speculating on the implications of biofeedback and real-time synthesized sound for player interaction and immersion.

The section on Technology opens with Spatial Sound for Computer Games and Virtual Reality by David Murphy and Flaithrí Neff.  The authors guide the reader through human spatial sound processing and the propagation of sound in space while pointing out the problems faced in transferring and accurately replicating these phenomena within computer game systems.  They conclude with a survey of the strengths and weaknesses of existing spatial sound technologies for virtual worlds, and look to future possibilities for computer game sound posed by the ongoing development of the technology.

Andy Farnell's chapter provides an in-depth critique of sample-based game audio followed by an analysis of the potential of procedural audio: the real-time design of sound.  For Farnell, audio samples have proven to be too limiting, both for the purposes of creativity in game sound design and for the promise of realism: audio samples are predicated upon selection whereas procedural audio is design. A close discussion of procedural audio techniques, both as they have been used and how they might be used to the benefit of computer games, leads the author to the conclusion that it is both pointless and wasteful of computer resources to pursue precise sonic realism: procedural audio can instead be used to provide just the necessary level of realism, a perceptual realism, that is required for the player to comprehend source and source behaviour whilst saving scarce resources for more interesting and immersive tasks.

Eoin Mullan's chapter delves further into the promise of procedural audio by providing a detailed exploration of the potential of physical modelling, a branch of procedural audio. He traces the technique's development from the synthesis of musical instrument sounds to its current state where it stands poised to deliver a new level of behavioural realism to computer games.  For Mullan, this will be achieved through the integration of the technology with game physics engines and through physical modelling's ability to provide unprecedented levels of player-sound interaction.

Current & Future Design is the subject of the final section.  First, Valter Alves and Licínio Roque present a lucid case for the importance of sound to the design and experience of computer games: the authors assert attention should be paid to sound from the start of the design process.  Alves and Roque discuss concepts such as resonance and entrainment as means to engage and immerse players in the gameworld. They present seven guidelines for game sound design and detail an illustrative example of the application of these heuristics.

Finally, Daniel Hug's chapter is a clarion call for a new aesthetic of computer game sound.  Through a discussion of pursuit of reality and cinematic aesthetics, two dominant paradigms in computer game sound discourse, it details the debt that game sound owes to cinema sound but then uses examples, in particular from many innovative game developers and from cinema's own subversive stream, to shrug off that mantle and argue for a new future for game sound design.  Rich in ideas and provocative in its discourse, the chapter is full of practical suggestions for making computer game sound not only a different experience to cinematic sound but an engaging and rewarding one too.

Closing the book is an appendix which is a lightly edited transcript of an online discussion forum to which the book’s contributors were invited. They debate and attempt to answer the question: What will the player experience of computer game sound be in the future?  This is, of course, an open-ended question and the unstructured, lively debate that ensues is indicative of the similarly open-ended potential for the future of computer game sound.

Whatever your need in picking up the book, I hope you will find it met.  At the very least, perhaps one of the contributions here will raise intriguing questions in your mind, an itch that will be scratched by future investigation on your part.  Perhaps the ideas contained within will inspire you to develop a new game sound design paradigm, to innovate technologically, or to push the frontiers of human-sound interaction. After all, the aim of this book is not just to contribute to the development of ever better computer games or more cogent analyses but it is also to cast an illuminating light on at least one part of humankind's relationship with sound, even as we step out of reality into virtuality.

Author(s)/Editor(s) Biography

Mark Grimshaw is a Reader in Creative Technologies in the School of Business & Creative Technologies at the University of Bolton, United Kingdom, where he runs the Emotioneering Research Group. He possesses an honours degree in music, an MSc in music technology, and a PhD in computer game sound from South Africa, England, and New Zealand and is widely published in the area of computer games, particularly on the topics of immersion and sound. Mark’s previous book was entitled The Acoustic Ecology of the First-Person Shooter and he is also the lead developer for WIKINDX, an Open Source, Virtual Research Environment in wide use around the world.