The hybrid world provides a framework for the creation of lessons that teach and test knowledge in the second language. In one of our example grammar and vocabulary lessons, the Thinking Head instructs the student to move a tiger to a lake—a student moving the toy tiger in the real world effectively moves a virtual tiger in the virtual arena. This type of interaction is beneficial for computer-based language learning especially because we are able to see a student has successfully understood the directions if the tiger is moved to the vicinity of the virtual lake. Physical movement helps the learner to internalise the novel relationships in the second language. We also provide for additional forms of communication, including dialogue with an embodied conversational agent and writing stories using markers on a whiteboard. In summary, our system provides a natural interface with which to communicate with the hybrid-reality learning environment.
TopIntroduction
In this chapter, we explore interactions with the Thinking Head computer interface arising from a fusion of inputs of voice, objects, and writing—without keyboard and mouse. The Thinking Head system for language learning teaches through stories; however, the Thinking Head is not just the storyteller, but rather the medium in which the storyteller and story exist, and through which a learner learns to tell the stories. A learner engages in the multisensory storytelling activity by interacting with physical props within a room to learn the stories.
A user manipulates real-world objects within the Thinking Head virtual arena. In our implementation, objects are most commonly small graspable things; however, our use of the term object applies more generally to refer to things of all sizes, persons, places, and ideas. Indeed, humans have given names to objects for thousands of years, and the meanings of objects take form as the words we use as tools in communication. In SIMULA 67, arguably the first object-oriented computer programming language, objects were representations in a discrete event system (Dahl, 1968). As Lorenz (1993, p. 12) characterises objects in computer programming, “An object is anything that models ‘things’ in the real world. These ‘things’ may be physical entities such as cars, or events such as a concert, or abstractions such as a general-purpose account.” Thus, for our purposes, an object is any natural thing or combination of things that can be quantified, described, or elaborated by man or machine.
Traditionally, computer multimedia applications consist of audio and video, while mulsemedia, or multiple sensorial media, enhances the user interaction experience. The use of realia (physical props such as apples, hats, and magazines) adds to the language learning experience because realia engages the senses. Object recognition determines the identity, location, and movements of known physical objects, and classification heuristics categorise unknown objects. Those objects are rendered with respect to landmarks in the virtual world, creating a hybrid media experience that enhances class-based learning and provides a strong practical component that is easily accessible in the classroom or at home. Finally, writing practice reinforces the lessons and adds more latitude for expressing creativity.
The first question we need to address is how to promote modes of mulsemedia learning beyond the standard keyboard and mouse interface. Although interactive systems challenge the brain, Crawford (2003) notes that the typical scenario for play with a computer works with most of the body immobile. The advantage of learning in a multisensory computer environment is that the user can retain the benefits of computer processing while interacting with tangible objects in the environment. The goal of language learning is to become functional in the real world. As it is impractical and expensive for many language learners to travel to the part of the world where people speak the target language, it is desirable to learn by oneself in an immersive and interactive environment.
Language is produced as a reflection of multisensory perceptions of the world around. As children, we naturally acquire language of our physical, social, and cultural environment, but as we grow older, the gaining of fluency in a new language takes a different route. Although some believe that it is more difficult for teens and adults to learn a language, Krashen (1987) argues that it is not that the older learners proceed more slowly, it is rather that the interactive and comprehensible nature of the environment of the younger learners favours their learning. It is common knowledge that adults simplify the complexity of the language when they speak with children. By gradually and incrementally increasing language complexity while maintaining comprehension, learners of all ages come to perceive language not simply as a logical collection of information but also as a system of understanding the world. We believe that when a learner engages in mulsemedia experiences that are challenging yet stimulating, the language that is learned is grounded in experience, providing basic language skills that are later necessary to engage in reading.