Platform Support for Multimodality on Mobile Devices

Introduction

Mobile devices have become an essential part of our daily life. They help us to fulfill various tasks like communicating with others, gathering data while working mobile, accessing information and content on the Internet or a company’s intranet, or scheduling business or private activities. One of the most important aspects of mobile devices in general is their vast heterogeneity; mobile devices are available in very different shapes and form factors belonging to a wide range of device classes like mobile phones, smartphones, PDAs, subnotebooks, or notebooks. This has far reaching implications on the interaction between the user and the mobile device. Each device is built of different technological components, providing different input and output capabilities (e.g., keyboard, numeric keyboard, pen, display, or speech) and communication interfaces (like WiFi, Bluetooth, etc.). Moreover, users may be in different situations while using mobile devices (e.g., in a noisy urban environment, environments with bad lighting conditions, at work or in-car with no hands free to use a keyboard). Thus, the way a user interacts with an application on mobile devices strongly depends on the user’s situation and preferences, the modalities and capabilities of the used device, and the type of application or task that should be performed.

Exploiting the available input and output capabilities of devices requires a set of functionalities that an application has to provide. These functionalities comprise the access and management of available input and output devices, the recognition, interpretation, and integration of different input streams, and the invocation of application functions due to particular user interactions. Furthermore, output has to be handled in terms of generation of output as well as styling and rendering of output according to the requirements of different output devices. Most of this functionality can be regarded as essential and thus, has to be considered in almost every multimodal application.

Instead of implementing this functionality from scratch for each new application, a generic multimodal platform can enable widespread reuse of common functionality, more efficient development in terms of time and cost, more stable and reliable implementations due to the reuse of tested code as well as standardization. Aspects such a platform would have to cope with include:

Heterogeneous input and output devices: Abstracting from the underlying, mostly heterogeneous hardware to allow reusable and cost-efficient implementation of components for multimodal interaction;

Arbitrary/Flexible combination of input and output modalities: Allowing the user to interact with multiple devices in parallel to achieve different levels of multimodality like parallel and/or complementary user input and output;

Seperation of application logic from user interactions: A clear separation of concerns by the provision of appropriate abstractions for the association of multimodal interactions with the internal logic and data model of an application supports the development of applications independent from particular considerations about interaction modalities.

Provision of general functionality for high reusability: Offering a standardized execution environment, that provides general functions for managing and controlling multimodal interaction components.

Support for distribution of components: Distributing components within a network of computing devices to exploit resources and capabilities of available devices to improve user interactions and to meet the components’ requirements on hardware resources;

A brief overview about current standards and standardization activities in the area of multimodality is given in Section 2. The Multimodal Interaction Working Group (MMI) of the World Wide Web Consortium (W3C) has created a number of official documents covering basic aspects of multimodal platforms. These documents are introduced as a foundation for the understanding of platform related issues of multimodal systems. Moreover, related work dealing with platform related issues is discussed.