Multimodal Human Computer Interaction and Pervasive Services

Multimodal Human Computer Interaction and Pervasive Services

Patrizia Grifoni (Istituto di Ricerca sulla Popolazione - Consiglio Nazionale delle Ricerche, Italy)
Indexed In: SCOPUS View 1 More Indices
Release Date: May, 2009|Copyright: © 2009 |Pages: 538|DOI: 10.4018/978-1-60566-386-9
ISBN13: 9781605663869|ISBN10: 1605663867|EISBN13: 9781605663876|ISBN13 Softcover: 9781616924805
Hardcover:
Available
$156.00
List Price: $195.00
20% Discount:-$39.00
TOTAL SAVINGS: $39.00
E-Book:
Available
$156.00
List Price: $195.00
20% Discount:-$39.00
TOTAL SAVINGS: $39.00
Hardcover +
E-Book:
Available
$188.00
List Price: $235.00
20% Discount:-$47.00
TOTAL SAVINGS: $47.00
OnDemand:
(Individual Chapters)
Available
$37.50
No Current Special Offers
TOTAL SAVINGS: $37.50

Description

The use of multimodality combined with mobile devices allows a simple, intuitive communication approach and generates new and pervasive services for users.

Multimodal Human Computer Interaction and Pervasive Services provides theoretical and practical concepts, methodologies, and applications used to design and develop multimodal systems. Collecting cutting-edge research by international experts, this Premier Reference Source addresses the many challenges of multimodal systems with a particular focus on mobile devices.

Topics Covered

The many academic areas covered in this publication include, but are not limited to:

  • Amibuity management in multimodal systems
  • Analyzing multimodal interaction
  • Mobile virtual blackboard
  • Multimodal human computer interaction
  • Multimodal input fusion strategies
  • Multimodal language processor
  • Multimodal user interface
  • Pervasive services
  • Technology acceptance models
  • Usability evaluation of multimodal interfaces
  • Web accessiblity to the visually impaired

Reviews and Testimonials

This book provides theoretical and practical scenarios, concepts, methodologies, standards, definitions and applications used to design and develop multimodal systems, focusing on mobile devices and pervasive services. It gives an overview of the existing works in this sector, discussing the different strategies adopted for the fusion input process, optimization processes on mobile devices, ambiguity and error handling related to one or more modalities, the fission process, the personalization and adaptation of multimodal mobile services and the accessibility and usability criteria.

– Patrizia Grifoni, Istituto di Ricerca sulla Popolazione - Consiglio Nazionale delle Ricerche, Italy

Table of Contents and List of Contributors

Search this Book:
Reset

Preface

This book provides a contribution on the theories, techniques and methods on multimodality and mobile devices for pervasive services. It consists of 24 chapters that provide an in-depth investigation of new approaches, methods and trends. Humans communicate using their five senses in a synergistic manner expressing key-concepts involving different modalities and/or two or more modalities simultaneously. Indeed, human-human communication involves in a synergistic manner several communication channels; it can use gesture, sketch drawing, handwriting, facial expressions, gaze and speech or their combination. Multimodal interaction systems can combine visual, auditory and other kinds of information in a flexible and powerful dialogue manner, enabling users to choose one or more interaction modalities. The use of multimodality combined with mobile devices allows a simple and intuitive communication approach everywhere and every time. The synergistic approach in using more than one modality to interact with computers - or more generally with other devices such as mobile devices – makes dialogue and communication flexible, natural and robust also for its closeness with the human-human one.

Effectiveness and naturalness of communication is particularly relevant for services. The great diffusion of mobile devices, along with the development of multimodal interaction, presents a new challenge for telecommunication companies and all organizations that can be involved in providing new services using mobile devices. One requirement for these services is that they and their information have to be accessible to every mobile situation. Even if in the last years several efforts have been carried out to provide computer and mobile devices interfaces with a similar flexibility, naturalness and robustness, a lot of theoretical and technological problems need to be faced when designing and implementing very natural, flexible and robust multimodal human computer interaction approaches. Naturalness, flexibility and robustness of interaction approaches are key elements to produce new and pervasive services for people. In developing multimodal pervasive services it is essential to consider perceptual speech, audio, and video quality for the optimum communication system design, the effective transmission planning and management respect to the customer requirements.

BOOK OBJECTIVE

The book aims to provide theoretical and practical scenarios, concepts, methodologies, standards, definitions and applications used to design and develop multimodal systems, focusing on mobile devices and pervasive services. It gives an overview of the existing works in this sector, discussing the different strategies adopted for the fusion input process, optimization processes on mobile devices, ambiguity and error handling related to one or more modalities, the fission process, the personalization and adaptation of multimodal mobile services and the accessibility and usability criteria. Moreover, the book contains some significant examples of pervasive multimodal mobile applications; it discusses as acceptance is the a basic condition for a wide use of each service, and it analyses transformations produced in some sectors, such as for example e-learning, using both, multimodal interaction and mobile devices.

AUDIENCE

Academics, researchers, technicians, students in computer science and all experts involved in designing multimodal and mobile pervasive services will find this book a valuable contribution to their knowledge and, they will find it a stimulus for new scientific activities on the important topics involved.

ORGANIZATION

The book contains 24 chapters organized in four sections. The first section introduces some of the basic concepts and describes the main theoretical and practical problems and features of multimodal interaction. The second section describes how some particular multimodal applications overcome design problems connected with combining different modalities and how Ontologies can be useful to improve interoperability of different environments and naturalness of multimodal interaction. Chapters contained in the third section consider methods and problems of multimodal interaction focusing on mobile devices, and considering their adaptability and usability features. This section introduces a discussion on changes on the communication and services fruition produced by using mobile devices in the particular scenario of e-learning.Finally the fourth section of this book contains chapters that describe standards, some guidelines and evaluation methods for multimodal systems.

The first section is opened by a study on people acceptance and adoption of mobile devices proposed by Judy van Biljon and Karen Renaud in the first chapter. Acceptance is the first step for the wide adoption and diffusion of a new technology. In particular, the chapter investigates the “mobile phone adoption by older users” proposing the novel Senior Technology Acceptance and Adoption Model (STAM), which incorporates acceptance factors into the adoption process. From chapter two to chapter nine the basic concepts, theories and technologies about multimodal interaction and multimodal systems are provided; these chapters give an overview of the lifecycle of information and data characterizing multimodal interaction. In particular, in chapter II Ferri and Paolozzi give an overview of the main problems, definitions and concepts related to multimodality, describing the “different approaches used in the multimodal interaction”, mainly addressing those approaches that use the new semantic technologies as well as the Semantic Web ones.

Naturalness of multimodal interaction depends on its feature of involving all five human senses simultaneously. D’Ulizia in chapter III provides an overview of the different approaches of the literature to combine and integrate different input modalities (fusion process). Based on state of art review on multimodal input fusion approaches, Chen and Sun in chapter IV present a novel approach to multimodal input fusion based on speech and gesture, or speech and eye tracking based on a combinational categorical grammar. This approach has been validated for mobile context.

“Naturalness and flexibility of the dialogue” can produce a very complex interpretation process and the presence of ambiguities can be a relevant problem to face. Caschera in chapter IV analyses approaches that have to cope issues connected to the interpretation process. She provides a classification of ambiguities into recognition, segmentation, and target ambiguities, and describes methods for solving different classes of ambiguities.

Naturalness of the multimodal interaction implies a symmetric communication between humans and machines (computers, mobile devices, and so on). For this reason when a user interacts with a multimodal system, s/he produces a multimodal input that is fused and interpreted. The multimodal system usually reacts and produces an output, which requires to be interpreted by the user. The output arrangement and organization (multimodal fission) is a focal point that is faced by Grifoni in chapter VI. This chapter provides some basic concepts involved in the fission process design and describes some of the most relevant approaches discussed in the literature, with a particular attention on the mobile devices use.

The problem of flexibility and capability to adapt functionalities on mobile devices to the users is faced in chapter VII by Esposito, Basile, Di Mauro e Ferilli. The chapter presents a two-phase construction of the user model: an initial static user model, and revised/adjusted model by considering the information collected by the user interaction with the device/context.

Designing and developing human machine interfaces is a critical task due to the fact that it is a time consuming and expensive task. This fact is particularly evident for multimodal environments. Coninx, De Boeck, Raymaekers and Vanacken with chapter VIII propose a Model-Based User Interface Design process (MBUID), in which the interface is de?ned by means of high level of abstraction, which gives the opportunity for creating intermediate prototypes and user evaluation, ultimately resulting in better and cheaper virtual environment interfaces.

This first section ends with chapter IX addressing the concrete aspect of the temporal and spatial organization of human pointing actions for multimodal human computer interaction, faced by Müller-Tomfelde and Chen. The role of pointing gestures combined with other modalities is presented and developers are provided “with substantial knowledge about the temporal-spatial organization of the pointing gesture”.

The second section of this book collects specific methodologies for designing and implementing multimodal systems. In chapter X Castellina, Corno and Pellegrino describe a multimodal system integrating speech-based and gaze-based inputs for interaction with a real desktop environment. The proposed approach shows how the combined use of auditive and visual clues permits mutual disambiguation in the interaction with a real desktop environment. Pittarello and Celentano in their chapter titled “Multimodality and Environment Semantics” provide a methodology devoted to identify the user-oriented features of the environment describing the physical environment at different levels of granularity, functionally and geometrically. A scene-independent ontology permits a level of interoperability between different environments. The chapter ends presenting a case study about the creation of a guided tour through the indoors and outdoors of the town of Venice using a multimodal web browser.

The importance in using an ontology for multimodal systems in different use cases has a further evidence in chapter XII “An ontology supporting an on-board vehicle multimodal interaction system”, by Pisanelli, Lazzari, Innocenti, Zanetti. In that chapter they introduce ontologies (Vehicle Security, Road and Traffic Security, Meteorological, Users' Profiles and Travel) supporting an on-board vehicle multimodal interaction system, which enables semantic interoperability for cooperation between the road infrastructure and assisting the driver.

Barricelli, Marcante, Mussio, Parasiliti Provenza, Padula and Scala in chapter XIII propose a methodology supporting a collaborative and evolutionary design of an interactive system. The proposed approach uses the metaphor of the “workshop of an artisan”. It starts from empirical experience to handle problems faced on the field, considering the need to interact everywhere and every time with mobile devices.

An important aspect that multimodality can face is connected with the possibility that it has to improve the information and services accessibility, and the possibility of any person to make use of them. Bacellar Leal Ferriera, Silva da Silveira, Gurgel do Amaral Leal Ferriera and Rodrigues Nunes in chapter XIV titled “Making the Web Accessible to the Visually Impaired” face the problem of making accessible information to blind persons. A study aimed at identifying and defining usability guidance compliant with accessibility W3C directives to facilitate the interaction between visually impaired people, and their access to web information and services, has been presented.

The potentiality to improve the access of people having different physical, psychological and cognitive attitudes to different information and services is one of the features of multimodal interaction that makes it very attractive for e-learning, and in particular for mobile learning. Spetch in chapter XV depicts the experience carried out in the European project RAFT. A widget based approach design for the user interface enables to build clients for a variety of hardware and devices in the learning environment, from mobile phones, PDAs, Tablet PCs, Section III discusses and details problems, features and applications of multimodal interaction focusing the attention in the case of mobile devices.

In chapter XVI Frattini, Corvino, Gaudino, Petriccione, Scotto di Carlo and Supino discuss how a speech driven interaction between user and service delivery systems may be the ideal solution for the development of ubiquitous and context aware applications; indeed, even if people have eyes and hands busy because they are making other activities too, they are usually able to interact using speech. The authors propose in their chapter an architecture for building mobile multimodal applications describing their experiences.

“A GRID approach to providing multimodal context-sensitive mobile social service” is presented in chapter XVII by Pierno, Scotto di Carlo, Magaldi, Russo, Supino, Romano and Bevilacqua. When designing a mobile information system, it is not only important input-output channels management and information presentation, but also context-awareness. The authors have described a platform that supports the development of multi-channel, multi-modal, mobile context aware applications, and its use in an emergency management scenario.

In chapter XIII Doyle, Bertolotto and Wilson depict an application providing mapping services and discuss the importance of opportunely identifying the visualization and interaction modes. Multimodal interfaces for mobile GIS represent a challenge according to the need to use them in different context and by different users, simplifying the inherent complexity of the majority of GIS interfaces. The chapter describes the multimodal mobile GIS CoMPASS, which uses a combination of speech and gesture input.

Using multimodal systems on mobile devices to access information and services can require to generate the user interface on-the-fly according to the different contents, devices and user’s needs. Chapter XIX, by Encelle, Baptiste-Jessel and Sèdes, presents an “Adapted Multimodal End-User Interfaces for XML-Content”. This system is characterized by a user interfaces for browsing particular XML content types that can be personalized; it is described in particular its use for visual impaired people.

The usability of mobile devices with their small screen is faced by Davcev, Trajkovik and Gligorovska in chapter XX, which describes a “Multimodal Mobile Virtual Blackboard” system. The system faces the addressed problem of adapting the application features to the specific user’s preferences and to the current use of the system. Adaptability is obtained by using XML agents and fuzzy logic.

The wide diffusion of cell phones, smart phones, Personal Digital Assistants and other mobile devices, and the possibility to design and to implement a very natural multimodal interaction is changing the approach to many individual and social activities. D’Andrea, Ferri, Fortunati and Guzzo in their chapter “Mobile devices to support advanced forms of e-Learning” describe that these transformation are producing a deep change in e-learning tools and in the environment in which e-learning takes place.

The forth section of this book discusses standards for multimodal applications, some guidelines to follow, in particular for visual representation of interaction for mobile multimodal systems. Finally usability evaluation methods for multimodal applications on mobile devices are discussed. Dahl in chapter XXII describes standards for multimodal interaction focusing the presentation on standards for voice and GUI interaction (W3C Multimodal Architecture, VoiceXML, SCXML, EMMA, and speech grammar standards).

Focusing on “visual representation of interaction for mobile multimodal systems” Deray and Simoff, in chapter XXIII present a framework for designing visual representations for mobile multimodal systems, suggesting moreover a set of design guidelines.

The final chapter of the book describes the relevant problem of usability evaluation for multimodal interfaces mainly for mobile applications and services. In this chapter Bernhaupt depicts a selection of “classical” methods for usability evaluation and introduces some new emerging methods for testing usability of multimodal interfaces with a particular focus on mobile devices.

    Patrizia Grifoni
    Istituto di Ricerche sulla Popolazione e le Politiche Sociali
    Consiglio Nazionale delle Ricerche, Italy

Author(s)/Editor(s) Biography

Patrizia Grifoni has been a researcher of the National Research Council of Italy since 1990, where she works within the staff researchers of MultiModal Laboratory of the Institute of Research on Population and Social Policies. She received her degrees in Electronics Engineering at the University of Rome “La Sapienza”. From 1994 to 2000 she was a professor of "Elaborazione digitale delle immagini" at the University of Macerata.

She is the author of more than 70 papers in international journals, books and conferences. She served as referee of several international conferences, books and journals and is co-organiser of international research workshops and events. Her scientific interests have evolved from query languages for statistical and geographic databases to human-computer interaction, multimodal interaction and languages, visual languages, visual interfaces, sketch-based interfaces, web technologies, social networks. She was responsible of several projects funded by Italian and International Institutions.

Indices

Editorial Board

Editorial Advisory Board
  • Augusto Celentano, Università Ca' Foscari di Venezia, Italy
  • Fang Chen, National ICT, Australia
  • Deborah Dahl, Conversational Technologies, USA
  • Fernando Ferri, Istituto di Ricerche sulla Popolazione e le Politiche Sociali -Consiglio Nazionale delle Ricerche, Italy
  • Irina Kondratova, NRC Institute for Information Technology - National Research Council, Canada

    List of the Reviewers

  • Augusto Celentano, Università Ca' Foscari di Venezia, Italy
  • Fang Chen, National ICT, Australia
  • Karin Coninx, Hasselt University, Belgium
  • Deborah Dahl, Conversational Technologies, USA
  • Fernando Ferri, Istituto di Ricerche sulla Popolazione e le Politiche Sociali - Consiglio Nazionale delle Ricerche, Italy
  • Irina Kondratova, NRC Institute for Information Technology - National Research Council, Canada
  • Marco Padula, Istituto di Tecnologia delle Costruzioni -Consiglio Nazionale delle Ricerche, Italy
  • Domenico M. Pisanelli, Istituto di Scienze e Tecnologie della Cognizione-Consiglio Nazionale delle Ricerche, Italy
  • Fabio Pittarello, Università Ca' Foscari di Venezia, Italy
  • Maurizio Rafanelli, Istituto di Analisi dei Sistemi ed Informatica "Antonio Ruberti"-Consiglio Nazionale delle Ricerche, Italy