Visual Languages for Interactive Computing: Definitions and Formalizations

Visual Languages for Interactive Computing: Definitions and Formalizations

Fernando Ferri (IRPPS-CNR, Italy)
Indexed In: SCOPUS View 2 More Indices
Release Date: August, 2007|Copyright: © 2008 |Pages: 532
ISBN13: 9781599045344|ISBN10: 1599045346|EISBN13: 9781599045368|DOI: 10.4018/978-1-59904-534-4


Visual languages are the defining component of interactive computing environments, yet in spite of the rapid pace of evolution of this domain, significant challenges remain.

Visual Languages for Interactive Computing: Definitions and Formalizations presents comprehensive coverage of the problems and methodologies related to the syntax, semantics, and ambiguities of visual languages. This all-inclusive reference authoritatively sets out the defining issues related to formalizing visual languages for interactive computing, creating a foundational reference for future research and application. As a unique collection of research on all critical elements of the field, this Premier Reference Source enables libraries to provide the defining research to drive the next generation of improvements to interactive computing.

Topics Covered

The many academic areas covered in this publication include, but are not limited to:

  • Customized Visual Computing
  • Designing a Visual Language
  • Documentation Methods for Visual Languages
  • Drawing Biochemical Networks
  • Extended Positional Grammars
  • Geovisualization and GIS
  • Interactive Visual Languages
  • Interface Design with UML
  • Management of Ambiguities
  • Multi-Facet Design of Interactive Systems
  • Multi-View Visual Languages
  • Questioning Usability
  • Semantic Web Service Composition
  • Sketch Understanding
  • Spatio-Temporal Databases
  • Theory of Visual Sentences
  • Visual and Diagrammatic Languages
  • Visual Data Mining
  • Visual Notation Interpretation and Ambiguities
  • Visual Programming Tool
  • Visual Query Languages
  • XML Data Management

Reviews and Testimonials

Academics, researchers, technicians, and students in computer science will find this book a valuable contribution to their knowledge of this important research area.

– Fernando Ferri, IRPPS-CNR, Italy

Table of Contents and List of Contributors

Search this Book:


Introduction from the Editor

This book gathers the contributions of authors working in the area of visual languages from countries as diverse as Spain, Italy, United Kingdom, Australia, Greece, Canada, Brazil, and the USA.

It consists of 22 chapters written by a total of 54 different authors to provide an in-depth investigation of new approaches and methods for visual language theory, visualisation techniques, aesthetic computing, and usability.

I hope that academics, researchers, technicians, and students in computer science will find this book a valuable contribution to their knowledge of this important research area.

In normal life, people organise a complex structure of signs to communicate with the world around them. The brain acts as an interface between the sign system and the environment, orienting the individual in space. Visual languages systemise the visual representation of concepts in a formalised language using visual objects such as images or pictorial objects and possibly formal visual expressions. They are based on simple visual elements, which construct symbols that can be grouped to form structured visual sentences.

The Overall Objective of This Book

The main objective and mission of this book is to present problems and methodologies concerning the syntax, semantics, and ambiguities of visual languages. Various syntactic techniques can be used to define visual languages, which differ significantly in the way they conceptualise a visual notation and describe its syntactic structure. From the semantic point of view, the precise meaning of a visual language is required in order to use it effectively. Finally, ambiguity is one of the main problems of visual languages. A visual configuration expressing a sentence or a visual action may be interpreted in a number of ways producing ambiguity. Visual languages offer an intuitive and incremental view of sentences, but may give different interpretations of the same sentence.


The book contains 22 chapters split into five sections. Section I examines the theory underlying visual languages. Section II discusses different approaches to and methods for visual languages on specific domains. Section III describes visual languages for the semantic Web. Section IV examines visual interfaces for standard languages. Finally, Section V considers several topics closely related to visual languages such as visualisation, aesthetic computing, and usability

Section I. Visual Languages Theory

Section I opens with a chapter by Bottoni, Costabile, Levialdi, and Mussio, introducing the Theory of Visual Sentences by highlighting their main concepts, presenting some problems central to the definition of a theory of visual languages, and describing current developments. After an informal overview of the main components of the theory, the authors explore the characteristics of interactive processes and the modelling of visual transformations by visual rewriting systems. They go on to examine the dynamics of visual sentences, focusing on the relationship between constancy and variability in visual sentences, enabling and disabling mechanisms and coordination of transformations. The chapter ends with a discussion of how the theory’s components can be integrated to provide a framework for the design of interactive visual systems before drawing some conclusions.

Chapter II, by Meyer and Bottoni, traces progress in Visual and Diagrammatic Languages: The Logic Perspective. The chapter starts by outlining the grammatical approach to visual language specification highlighting its shortcomings as a basis for this research plan. It then revisits the history of logic approaches to diagrammatic languages and details the comparative advantages and disadvantages of the different types of logic formalisation. Finally, it develops a new approach based on linear logic, which avoids most of these shortcomings and completely subsumes the grammatical approach.

Chapter III, by Bottoni, Frediani, Quattrocchi, Rende, Sarajlic, and Ventriglia, proposes a Transformation- Based Metamodel Approach, which enables an abstract view of the semantic roles that visual elements can play with respect to the process being described. More specifically, the chapter proposes an integrated framework and interactive environment based on a collection of metamodels in which it is possible to express both syntactical characterisations of diagrammatic sentences and their semantic interpretations.

Chapter IV, by de Lara and Guerra, presents an approach for the definition of Multi-View Visual Languages (MVVLs). These are made up of a set of different diagram types, which are used to specify the different aspects of a system. The chapter introduces two techniques to define MVVL environments: Meta modelling and graph transformation. The former is used to describe the syntax of the language as a whole. A meta-model for each of the language’s diagram types (viewpoints) is defined as a restriction of the complete MVVL meta-model. Consistency between views is ensured by translating each one into a unique repository model, which conforms to the meta-model of the language as a whole. The translation is performed by automatically generated graph transformation rules.

Chapter V, by Costagliola, Deufemia, and Polese, presents an overview of eXtended positional grammars (XPG), a grammar formalism for modelling visual notations, which represents an extension of context-free grammars and describes the XpLR parsing methodology. XPG and XpLR extend positional grammars (PG) and the associated pLR parsing methodology. These extensions have enabled the modelling and efficient parsing of a wide class of notations. The associated parsing algorithm is the XpLR parser based on the well-known LR parsing technique. The benefits of the formalism of a visual language include easier customisation and modification as well as maintenance and debugging, code and report generation by definition of suitable semantic productions, and implementation of visual and textual languages within a common framework.

Chapter VI, by D’Ulizia, Grifoni, and Rafanelli, classifies the Different Kinds of Ambiguities that can arise in visual sentences, distinguishing between lexical and syntactic ambiguities. When an image associated with a visual sentence is unable to express exactly (be completely faithful to) the user’s intentions, the system may produce an erroneous interpretation. Ambiguities are generally produced by (1) the language, which can produce such one-to-many relationships, (2) imprecision introduced by theinteraction behaviour producing the visual sentence. In the first case, an image can assume more than one meaning. The second case is connected with incorrect/imprecise information that does not permit an unequivocal interpretation of the image produced by the drawing actions.

The final chapter in this section, by Caschera, Ferri, and Grifoni, proposes Different Kinds of Solutions to Ambiguities that can arise in visual sentences. This chapter deals with ambiguities related to the system’s interpretation function and methods to resolve them. These methods can be grouped in three classes: prevention of ambiguities, a-posteriori resolution, and approximation resolution methods. Prevention methods consider only a predefined set of possible system configurations, avoiding ambiguous configurations. A-posteriori resolution methods are based on mediation techniques, which enable the user to disambiguate his or her intention by dialogue. In contrast, approximation resolution methods are based on theories such as Fuzzy Logic, Markov Random Field, and Bayesian Networks and do not require user disambiguation.

Section II. Approaches and Methods for Specific Domains

Section II opens with the chapter by Caschera, D’Ulizia, and Tininini, which analyses the main characteristics of VQLs, concentrating on visual languages to Query Conventional Relational Databases but also examining information systems with a less rigid structure such as Web resources storing XML documents. It considers two main VQL classifications: the adopted visual representation technique (e.g., based on forms and tables, diagrams, icons, sketches, or combinations thereof) and the underlying data model (e.g., visual languages to query relational databases, object-oriented databases, collections of XML documents, and languages specifically designed for particular data such as geographic and multidimensional data).

Chapter IX, by Cavalcanti, Schiel, and de Souza Baptista, focuses on the specific category of Visual Query Languages for Spatio-Temporal Databases, which enable formulation of queries involving both spatial and temporal dimensions. Current papers treat these dimensions separately with only a few integrated proposals. This chapter presents a VQS called spatio-temporal visual query environment (STVQE), which allows the formulation of conventional, spatial, temporal, and spatio-temporal database queries in an integrated environment. With S-TVQE, the user, instead of querying the database by textual query languages, interacts with the system by visual operators to state the query conditions.

Chapter X, by Fogli, Marcante, Mussio, Provenza, and Piccinno, considers that the knowledge relevant to the design of an interactive system is distributed among several stakeholders: domain experts, software engineers, and human-computer interaction experts, and presents a Multi-facet Design of Interactive Systems. Each community describes an interactive system through visual sentences of a visual language (VL). This view results in an approach to VIS design based on the definition and use of three visual languages. Each VL permits user-system interaction process to be specified from a different point of view and for a different audience.

The chapter by Deray and Simoff starts from the assumption that although interaction is the core of interactive computing, its mechanisms remain poorly understood. The tendency has been to examine interactions in terms of the results they produce rather than to provide mechanisms explaining how interactions unfold in time. The authors present a framework for creating visual languages to represent interactions using human movement as a source for the language’s core concepts. The approach is motivated and supported by evidence from research on kinaesthetic thinking that constructs based on human movement support higher-level cognitive processes and can be intuitively recognised by humans.

The final chapter of Section II, by Deufemia, considers Sketch Understanding. This is a particularly difficult task as the symbols of a sketched diagram can be drawn with different stroke orders, numbers, and directions. The recognition process is often made even harder by lack of precision and ambiguities in messy, hand-drawn sketches. The chapter presents a brief survey of sketch understanding techniques and tools.

Section III. Visual Languages for the Semantic Web

Chapter XIII, by Catarci, Kimani, and Lodi, examines User-Interface Formalisation in Visual Data Mining. A formal specification facilitates the description of the system properties without touching on implementation details, and enables fundamental design issues to be detected before they are manifested in the implementation. An approach for the formalisation of the visual interface of a core data mining system is given.

Chapter XIV, by Kontopoulos, Bassiliades, and Antoniou, discusses the Visualisation of Defeasible Logic Rules in the semantic Web domain. Logic plays an important role in the development of the semantic Web and defeasible reasoning seems a very suitable tool. However, it is too complex for end users who often need graphical traces and explanation mechanisms for the derived conclusions. The chapter proposes an approach that uses directed graphs to assist the user by offering the notion of direction, which appears highly applicable to the representation of rule attacks and superiorities in defeasible reasoning.

The last chapter in the section, by Vrakas, Hatzi, Bassiliades, Anagnostopoulos, and Vlahavas, is concerned with Knowledge Representation for AI Planning Problems, especially those related to semantic Web service composition. It discusses current approaches in encoding planning problems and presents ViTAPlan, a user-friendly visual tool for planning.

Section IV. Visual Interfaces for Standard Languages

Chapter XVI, by Almendros-Jimenez and Iribarne, shows how to use and specialise UML Diagrams for Describing the User Interfaces of a software system considering three specialised UML diagrams called user-interaction, user-interface, and GUI-class diagrams. It also examines code generation to implement the system’s user interfaces through GUI-class diagrams and user-interaction diagrams. A case study of an Internet book shopping system is introduced to test and illustrate the proposed user interaction and interface design technique.

Chapter XVII, by Campi, Martinenghi, and Raffio, describes a visual framework, XQBE that covers the most important aspects of XML Data Management, spanning the visualisation of XML documents, query formulation, the representation, and specification of document schemata, the definition of integrity constraints, the formulation of updates, and the expression of reactive behaviours in response to data modifications.

Section V. Visualisation Aesthetic Computing, and Usability

Chapter XVIII, by Voudouris and Marsh, discusses how cartography, GIS, and HCI shape aspects of Geovisualisation can support decision-making. The chapter analyses the relationships among geovisualisation, human computer interaction (HCI), geographic information systems (GIS), and cartography as a means of supporting decision-making. It emphasises the importance of data modelling and associated visualisations in terms of what the user can do by way of analysis and the methods by which he or she can undertake the analysis.

The following chapter, by Skhiri and Zimanyi, presents a Graph Layout Algorithm for Drawing Biochemical Networks. Due to the huge amount of information available in biochemical databases, biologists need sophisticated tools to accurately extract the information from such databases and interpret it correctly. Those tools must be able to dynamically generate any kind of biochemical subgraph (i.e., metabolic pathways, genetic regulation, signal transduction, etc.) in a single graph. The visualisation tools must be able to cope with such graphs and take account of the particular semantics of all kinds of biochemical subgraphs. The chapter presents an algorithm designed for the generic representation of biochemical graphs, in which users can present knowledge about how to draw graphs in accordance with the biochemical semantics.

Fishwick discusses the Aesthetic Computing Method. The purpose of aesthetic computing is to apply the theory and practice of art and design to the field of computing. The range of aesthetics within the arts is broader than in mathematics and computing where aesthetics is often synonymous with optimality criteria. This chapter introduces aesthetic computing for the multimedia representation of formal structures.

Chapter 21, by Costa, Grings, and Santos, presents Documentation Methods for Visual Languages. Visual programming languages (VPL) are generally self-documenting; this chapter analyses their use for the design and documentation of real world applications. Finally, the chapter proposes new tools and approaches for the documentation of data flow for VPL.

The book ends with the chapter by Padula and Reggiori, discussing Usability. This does not propose a specific viewpoint on usability but rather examines numerous concepts, aspects, and potentialities, which must nowadays be taken into consideration when detailing usability and design suited systems. This topic is currently of great interest in the field of human-computer interaction as it is highly dependent on the user interface. Systems must be used by a community of users in their working activity as a whole to process material or information, modifying not only rough material but also the working environment and methods. This requires them to be considered as tools in a social context, which expects ever-greater technological progress.

Author(s)/Editor(s) Biography

Fernando Ferri received his degrees in Electronics Engineering and PhD in Medical Informatics at the University of Rome ‘La Sapienza’ Actually Researcher at the National Research Council of Italy, he was a Contract Professor from 1993 to 2000 of ‘Sistemi di Elaborazione’ at the University of Macerata.

He is the author of more than 90 papers in international journals, books and conferences. His main methodological areas of interest are Human-Computer Interaction Visual Languages, Visual Interfaces, Sketch-based Interfaces, and Multimodal Interfaces, Data and knowledge bases, Geographic Information Systems.