Evaluation of Spatio-Temporal Microsimulation Systems

Introduction

Modeling individual movement behavior is a complex task and requires thorough validation throughout the modeling process. The complexity of spatio-temporal microsimulation systems originates mainly from the vast solution space of individual movements in geographic space and time. The size of the solution space is closely linked to the spatial and temporal resolution of the microsimulation, which is typically predetermined by the application. The outcome of a microsimulation is a complete mobility model for a given region, time period and population. Based on a synthetic population it provides a detailed schedule about who moves when and where using which mode of transport. This comprehensive information makes the validation of spatio-temporal microsimulation systems challenging.

In current practice the evaluation of microsimulations relies on a partial evaluation of mobility characteristics. Most often a comparison with traffic counts is performed as e.g. in (Gao et al., 2010; Horni et al., 2009; Meister et al., 2010). Traffic counts have the advantage that they can be comparably easy obtained. However, traffic counts do not contain origin-destination information and are typically available for vehicular traffic only. Thus, they cover only a small aspect of individual movement. In addition, most traffic counts are available for major roads only and are therefore not representative for the whole street network. In other words, traffic counts are a vital source for the validation of microsimulation systems but their application is limited to a subset of model characteristics.

The example of traffic counts illustrates that a new holistic validation concept for spatio-temporal microsimulation systems is needed. Only a validation which considers a broad set of mobility characteristics can ensure that the outcome of a microsimulation is a truthful reproduction of reality. The required variety of validation data to implement such a concept is just becoming available due to the advancement of information and communication technology. Thus, we are at the right moment of time to animate the discussion about a new validation standard. In addition, research on spatio-temporal data mining and analysis has made tremendous advances during the past decade. As a result, a rich set of preprocessing, feature extraction, indexing and data mining methods are available to exploit and handle spatio-temporal data.

The validation of microsimulation systems is a manifold task. In general, the validation workflow of the modeling process can be divided into three parts, namely input data validation, internal model validation and external model validation (see Figure 1). Input data validation ensures that the model is build using high-quality data. Usually, the provided input data comes from secondary data sources and has originally been collected for a different purpose. For this reason it is important to show in a first validation step that the input data is an appropriate source for the goal of the overall modeling process. Internal model validation measures how well the model can predict its input data, i.e. how reliable a predictive model will perform on (unseen) training data. A high internal quality is a prerequisite for the external quality of the model. If the input data of the model is difficult to predict, this is likely to be the case also for unknown data. In addition, as microsimulation systems typically rely on non-deterministic algorithms, the variability of the model is subject to internal model validation. Finally, during external model validation the model output is compared to either independent real world data or other model results with known high quality. This step ensures the final quality of the model results. If the model outcome meets a given quality standard the process of modeling is finished. Otherwise, either the input data or the modeling process are subject to change and to a repeated validation, leading to a validation cycle.

Figure 1.

Quality cycle in microsimulation systems