People perceive information from the outside world using several sensory organs rather than through a single one. First, relevant information is processed in the corresponding sensory area independently and then integrated in a multi-sensory processing cerebral region. The integration process influences the later behavioral responses to the sensory stimuli. Based on previous research, this chapter summarizes the neural mechanisms and the influencing factors of audiovisual integration. These include the relationship between the spatial location of auditory stimuli, visual stimuli, and verbal perception. According to research in neurology, the main neural mechanisms of audiovisual integration are from the superior temporal sulcus in the rear left region of the human brain. In the future, research on the combination of virtual reality technology and audiovisual integration should be continued to enable the research achievements to have more ecological effects.
TopIntroduction
In daily life, people perceive information from the outside world not by one single sensory stream, but by several sensory streams that encode and further process the information in a special area of brain. For example, when enjoying the color, aroma and taste of something delicious, this refers to the information process of the senses of sight, smell and taste. The effective integration and processing of sensory streams from various stimuli is named multisensory integration (Ernst & Bulthoff, 2004). Many studies have reported that information from different sensory modalities interact (Alais & Burr, 2004; G.A. Calvert, Campbell, & Brammer, 2000; Giard & Peronnet, 1999; Hershenson, 1962; Macaluso, Frith, & Driver, 2000; McGurk & MacDonald, 1976; Molholm et al., 2002; Schroeder & Foxe, 2005; Shipley, 1964). Stein, B.E. et al. proved the existence of the multisensory integration effect in 1996. Specifically, they stated that a single visual event is perceived as being brighter when accompanied by an auditory signal than when presented in isolation (Stein, London, Wilkinson, & Price, 1996). Multisensory integration is a hot area of study in recent research in neurology, and one in which audiovisual integration is the most extensive point.
In early ethology research, the discovery of the McGurk effect is considered to be a landmark event in audiovisual integration research (McGurk & MacDonald, 1976). The study found that an integrated influencing effect existed between visual stimuli and auditory pronunciation. For example, when the visual stimuli is the face of a person who is pronouncing the syllable “ga” and the auditory stimuli is the syllable “ba”, the answer of participant is “da”. Since then, neural mechanisms of audiovisual integration have been extensively studied (Andres, Oram Cardy, & Joanisse, 2011; Raij, Uutela, & Hari, 2000; N. van Atteveldt, Formisano, Goebel, & Blomert, 2004; N. M. Van Atteveldt, Formisano, Blomert, & Goebel, 2007). Research in recent years has shown that mechanisms of audiovisual integration are mainly influenced by the order and the locations of stimuli (Senkowski, Talsma, Grigutsch, Herrmann, & Woldorff, 2007; Stevenson & James, 2009). The results also indicated that the region of the brain which regulated mechanisms of audiovisual integration is located in the superior temporal sulcus (Alsius, Navarra, Campbell, & Soto-Faraco, 2005; G.A. Calvert, 2001; G.A. Calvert et al., 1999; G. A. Calvert et al., 1997; Hertrich, Dietrich, & Ackermann, 2011; Navarra, Alsius, Velasco, Soto-Faraco, & Spence, 2010). The region regulates the neural activity of a specific pathway by feedback. Based on the previous research, this paper summarizes and comments on neural mechanisms and the influencing factors of audiovisual integration.