Multimodality and Environment Semantics

Introduction

Proper access to urban environments like streets and squares, buildings, parks, museums, offices, etc., and correct movement inside them are not always simple and straightforward tasks. They are conditioned by a variety of circumstances, biased by the abilities of the visitors and by the complexity of the environments, and raise interesting issues about design and usability of help systems and information points for assistance to people in public places. Physical and cognitive deficiencies could prevent some persons from easily recognizing locations, objects and their relations. Such a situation often leads the visitors of an environment to be unable to get information about objects of interest, to recognize dangerous situations, and to profit from support tools for moving safely, like parapets, traffic-lights, pedestrian crossings, etc.

In this work we discuss the problem of describing complex human environments from a semantic perspective, making explicit the meaning of the environment elements and their relation with the persons moving inside it. We propose a methodology and an implementation architecture to design and build systems able to provide meaningful information to assist people to orient and move. Different persons with different skills need to receive different levels of assistance: for example, elderly people with visual and cognitive deficiencies require proper communication styles and devices, and need more assistance than people with normal physical and cognitive faculties.

Solutions exist, such as GPS based navigators, for guiding people through unknown places to destination, helping them to follow a reasonable path. However, most of them provide a simplified map and suggest direction changes at proper places with standard messages related to streets and roads only, such as “turn left after 50 meters”, “go straight”, “turn right at the end of the street”, and so on. They do not interpret the route in terms of how the user perceives the surroundings, what environment components can be meaningful as landmarks for a visitor, what is their purpose, as a high-level description of the environment could provide. For example, a semantic oriented navigator could suggest directions to a person walking in a town with terms such as “at the traffic-light turn left”, or “take the road behind the fountain”. In a GPS navigator only a few landmarks, the so called Points of Interest (POI), are marked by type or by touristic relevance and are shown on the map, but they are not used in composing the vocal instructions.

The lack of a complete high-level description of the environment seen by the walker is a consistent limitation that prevents conventional navigation systems to assist the users to identify the relevant objects in the environments as reference points for moving and finding directions, even if this is the usual way humans exchange information when asking for assistance. Indeed, most of the solutions available are unable to contribute to the creation of a mental model of the territory; providing the instructions to reach a destination as a sequence of steps, they miss the big picture in favor of a local perspective.

In addition to providing an environment description based on semantic related concepts, the association of audio signs to different locations could help the user to mark them more clearly and to progressively build an audio, as well as a visual, map of the visited places. Audio marking can be very useful to enable a sudden recognition when the user re-enters the same place.