Article Preview
TopIntroduction
As mobile touchscreen technology has become more widespread, there have been many technological advances but one key feature remains the same; touchscreen mobile devices often use cut-down versions of desktop user interfaces placing high demands on the visual sense that may prove awkward in mobile settings. Mobile device users could be considered to be situationally impaired because it is possible to lose the use of different senses temporarily at different times, locations and other situational contexts. Currently, many available devices support audio and tactile feedback for simple alerts such as incoming call notifications through the use of standard built-in vibrotactile actuators and audio speakers. These may be leveraged to provide user feedback to different sensory modalities when the visual sense is overloaded or unavailable.
Mobile phones are personal devices, always on and always with us, which means that whether it is in our bag or pocket, or we are in a meeting, at a party, or listening to music, we still want to be able to interact with our device. In these situations, visual feedback is not always appropriate. Although a user’s eyes may be busy focusing on the primary task, many activities do not otherwise restrict users from attending to information using their remaining available senses. This is when multimodal interaction is of benefit so that, for instance, messages can be presented through the audio modality and warnings can be presented through the tactile. Unfortunately, these modalities can also be inappropriate at times. For example consider this typical usage scenario: Sam is walking to a meeting with her mobile phone in her bag when she receives an important calendar reminder. As her phone is not in contact with her body, a tactile alert would probably go unnoticed so the reminder would be best presented in audio. Next, Sam boards a train to continue her journey and downloads some music for her phone. Given that the train is noisy and the phone is in her pocket, audio alerts alone would be insufficient to inform her of her completed download. At the same time, tactile alerts would be slightly masked, as the phone is not in direct contact with her skin. At this time, a combination of audio and tactile feedback could let her know when her song has been downloaded. Finally, Sam arrives at her meeting and receives an urgent email from her husband. It would be rude for Sam to disrupt the meeting with audio feedback informing her of the incoming email. In this case, a tactile cue would be much more subtle and socially acceptable. This scenario is an example of the need for mobile devices to provide alternative presentation modalities through which information may be presented if the context requires.
As mentioned, multimodal feedback is often used to reduce the visual load on mobile device users. The possibilities of communicating information and enhancing interaction through senses other than vision, e.g., sound and touch, has generated a rich body of research (Brewster, 2002; Cockburn & Brewster, 2005; Fukumoto & Sugimura, 2001; Gaver, 1987; Hall, Hoggan, & Brewster, 2008; Kaaresoja, Brown, & Linjama, 2006; Lee & Zhai, 2009; Mereu & Kazman, 1996; Poupyrev & Maruyama, 2003). The existing research has demonstrated that audio and tactile feedback can be beneficial to mobile touchscreen users, increasing typing speeds and reducing errors with some training. However, as the scenario above has demonstrated, users need to be able to switch effortlessly between different modalities depending on the situation. Users also need the option of several different modalities. Much of the research so far does not give the user a choice of modalities but simply provides one output modality, resulting in unimodal interaction.