Designing Mobile Multimodal Applications

Designing Mobile Multimodal Applications

Marco de Sá (University of Lisboa, Portugal), Carlos Duarte (University of Lisboa, Portugal), Luís Carriço (University of Lisboa, Portugal) and Tiago Reis (University of Lisboa, Portugal)
DOI: 10.4018/978-1-60566-978-6.ch005
OnDemand PDF Download:


In this chapter we describe a set of techniques and tools that aim at supporting designers while creating mobile multimodal applications. We explain how the additional difficulties that designers face during this process, especially those related to multimodalities, can be tackled. In particular, we present a scenario generation and context definition framework that can be used to drive design and support evaluation within realistic settings, promoting in-situ design and richer results. In conjunction with the scenario framework, we detail a prototyping tool that was developed to support the early stage prototyping and evaluation process of mobile multimodal applications, from the first sketch-based prototypes up to the final quantitative analysis of usage results. As a case study, we describe a mobile application for accessing and reading rich digital books. The application aims at offering users, in particular blind users, means to read and annotate digital books and it was designed to be used on Pocket PCs and Smartphones, including a set of features that enhance both content and usability of traditional books.
Chapter Preview


The advances of mobile devices and technology have permitted their inclusion on our daily activities, supporting and propelling productivity and communication on several levels. Moreover, the ever‐growing diversity and potential of these devices extends their usage through multiple settings, accentuating their ubiquitous and pervasive nature. One of the major contributions to this is undoubtedly the multiplicity of interaction possibilities. These open opportunities for the introduction of multimodal applications that can offer new levels of usability and adequateness to the multiple scenarios in which mobile devices play key roles. In fact, within ubiquitous activities, multimodal interfaces are paramount. They provide alternatives to usual interaction streams that can cope with the challenges that emerge in different occasions. Users can adapt interaction to the context of use by selecting the interaction modalities that are more adequate to the current situation [0]. On another dimension, multimodalities also enlarge the target audience of mobile applications. They provide the access impaired users require to common tools and information. Moreover, the mobility and multimodal dimensions pave the way to the design of new applications that aid them greatly while completing their daily tasks.

As usual, these advantages do not come without drawbacks. In fact, multimodal applications and interfaces, used in mobile contexts by an extended audience, are susceptible to (1) more environmental variables, such as noise; (2) extra device characteristics and diversity, for instance the availability of a camera or microphone; and (3) a myriad of user abilities or impairments, such as normally sighted or visually impaired users. Accordingly, besides the existing challenges that hinder the design process of mobile applications (e.g., device’s size, lack of keyboard/mouse, screen resolution, mobile use, frequently as an accessory to parallel activities), multimodal applications require specific attention to the surrounding context of use and how it affects usability. Moreover, allying this with the mobility and infinite usage contexts that mobile devices provide, the design process becomes extremely demanding and challenging.

In order to mitigate these difficulties and to cope with the added challenges that designing mobile multimodal applications (MMAs) brings, the design process needs to be carefully addressed. First and foremost, the process has to be taken out of the lab, into the real world. Only there designers and users can grasp the details and problems that affect interaction and the usability of MMAs. Then, to capture these difficulties at the early design stages, prototyping and evaluation must be adjusted to the complexity of the envisaged contexts and modes of interaction. Evaluation sessions of mobile applications generally require testers and designers to follow users during tests on the real world, leading to privacy issues and biased results, also proving to be difficult to conduct. Finally, the selection of proper locations and scenarios for evaluation must be considered.

This chapter introduces a set of studies that address the added difficulties of designing multimodal applications and user interfaces for mobile devices. The first contribution that resulted from these studies is a scenario generation and context selection framework that aims at supporting the design of MMAs and their evaluation in-situ. It offers designers a set of guidelines that focus details that are paramount during the design and evaluation of mobile and, in particular, multimodal applications. The framework delineates variables that can affect the usability of MMAs and offers a systematic approach to the selection and generation of scenarios that cover a wide set of details and transitions between contexts and settings which are commonly overlooked.

The second contribution that emerged during these studies is MobPro, a prototyping and evaluation tool which enables designers and users (with no programming experience) to quickly create mixed‐fidelity prototypes of multimodal applications. Furthermore, it supports the ability to evaluate them in‐situ, while using actual mobile devices. The tool comprises a prototype designer, a mobile runtime environment and an evaluation/analysis tool.

Complete Chapter List

Search this Book: