Abstract
Perception in the real world is inherently multisensory, often involving visual, auditory, tactile, olfactory, gustatory, and, on occasion, nociceptive (i.e., painful) stimulation. In fact, the vast majority of life’s most enjoyable experiences involve the stimulation of several senses simultaneously. Outside of the entertainment industry, however, the majority of virtual reality (VR) applications thus far have involved the stimulation of only one, or at most two, senses, typically vision, audition, and, on occasion, touch/haptics. That said, the research that has been conducted to date has convincingly shown that increasing the number of senses stimulated in a VR simulator can dramatically enhance a user’s ‘sense of presence’, their enjoyment, and even their memory for the encounter/experience. What is more, given that the technology has been improving rapidly, and the costs associated with VR systems are continuing to come down, it seems increasingly likely that truly multisensory VR should be with us soon (albeit 50 years after Heilig, 1962, originally introduced Sensorama). However, it is important to note that there are both theoretical and practical limitations to the stimulation of certain senses in VR. In this chapter, after having defined the concept of ‘neurally-inspired VR’, we highlight some of the most exciting potential applications associated with engaging more of a user’s senses while in a simulated environment. We then review the key technical challenges associated with stimulating multiple senses in a VR setting. We focus on the particular problems associated with the stimulation of the senses of touch, smell, and taste. We also highlight the problems associated with the limited bandwidth of human sensory perception and the psychological costs associated with users having to divide their attention between multiple sensory modalities simultaneously. Finally, we discuss how the findings provided by the extant research in the cognitive neurosciences might help to overcome, at least in part, some of the cognitive and technological limitations affecting the development of multisensory VR systems.
If the virtual reality apparatus, as you called it, was wired to all of your senses and controlled them completely, would you be able to tell the difference between the virtual world and the real world?
What is real? How do you define real? If you're talking about your senses, what you feel, taste, smell, or see, then all you're talking about are electrical signals interpreted by your brain.
From ‘The Matrix’ (1999)
TopIntroduction
One of the main aims of virtual reality (VR) is to make people feel ‘real’ what is actually not really there. However, just as the above quotation suggests (see also Haden, 2005), the very question to be addressed here is what is ‘real’? In our culture, we take for granted the fact that the physical world is real and continues to exist regardless of whether we observe it or not. However, from a psychological standpoint, feelings and sensations are only real for a given observer (they do not, in themselves, exist); That is, perceptual ‘facts’ only exist within our minds/brains. Electrochemical signals are processed by our brains and this results in percepts that might, or then again just might not, have a counterpart in the external world (note that perceptual illusions constitute just one well-known example of ‘a perceptual reality’ that does not correspond to an actual external stimulus; e.g., Gregory, 1966, 1967). On the basis of such considerations, it soon becomes clear that in order for VR simulators to effectively create environments that “feel real”, they should try to reproduce the same neuronal activation as that generated by external stimuli. This is an important point because it suggests that the aim of VR is not necessarily to seek the closest correspondence possible between the qualities of real and virtual objects, but rather between the neural activation that is generated by real objects and events and that generated by virtual objects and events. Consequently, VR should be based, to a far greater extent than many people realize, on what we know about the ways in which our brains process incoming sensory signals (i.e., an approach that can be defined as ‘neurally-inspired’ VR). In this chapter, we focus on the relationship between the development of VR systems and one key attribute of the way in which information is processed by the human brain: Namely, the integration of signals generated by different sensory systems that leads to the construction of the multisensory perceptual experience of objects and events in the external world.
VR has come a long way over the last few decades (Laurel, 1995). However, the majority of research has been devoted to enhancing the quality/resolution of VR in just one sensory modality, namely vision (e.g., Burdea & Coiffet, 1994). It is, however, important to note that we perceive with all of our senses, and not just with our eyes (no matter whether we are aware of it or not). Audition, touch, taste, and smell (just to cite a few of the many senses; see Durie, 2005) all contribute to our everyday multisensory experiences (see Calvert, Spence, & Stein, 2004, for a review). Indeed, it has been argued that the majority of life’s most enjoyable experiences are inherently multisensory (e.g., Spence, 2002). Think only of a great meal. The food must not only look good. The smell, temperature, taste, and even the sound (yes, the sound; see Zampini & Spence, 2004) need to be appropriate as well in order to create a truly memorable dining experience (though note that striking incongruity can also be memorable). Actually, even if we are not fully aware of it, our brains likely judge ‘all’ experiences in terms of a perceptual ‘whole’ (what is known as multisensory Gestalt; e.g., Gallace & Spence, in press; Spence, Sanabria, & Soto-Faraco, 2007) rather than on the basis of a single channel of information.