ISEQL, an Interval-based Surveillance Event Query Language

Introduction

For the detection of certain behavioral or other patterns low-level events are usually not very useful. For example, when trying to identify the exchange of a package or an item of luggage at a train station using a video surveillance system, we have hundreds or thousands of people carrying items appearing in the video footage. Consequently, just checking the presence of persons and items is not sufficient, we are interested in specific sequences of events, such as person A carrying item X, setting it down and leaving, and finally person B picking up X. Deriving knowledge on a higher level from low-level events by combining the latter to complex structures is the task of an event query language (EQL).

An issue of many current EQLs is the lack of formal semantics (Eckert, 2011). In our approach we propose ISEQL, a language based on relational algebra for detecting high-level events. The relational algebra is clearly defined and forms the basis for many database query languages such as SQL. Additionally, this allows us to tap into the results of database research, such as optimizing the execution of relational algebra expressions by rewriting them. Relational databases are also widespread, which means that developers and users who are familiar with these systems have no problems in understanding the basic concepts of our language.

Using relational algebra as a foundation for an EQL is not a completely new idea (Eckert, 2011; Arasu 2006). The novelty of our approach is mapping the concepts of event detection to a relational algebra extended by interval operators based on Allen’s interval relationships (Allen, 1983). More and more database vendors integrate interval or range types and operators into their systems (PostgreSQL manual; Temporal tables), which allows us to re-use some of this functionality. These operators make it much easier to handle temporal constraints directly instead of modeling them in the traditional relational model. In summary, we make the following contributions:

• •

  We define an Interval-based Surveillance Event Query Language (ISEQL) based on relational algebra extended by interval operators.

• •

  We illustrate how ISEQL can be used to detect high-level events in a video surveillance context.

• •

  We describe in detail the framework developed for event detection.

• •

  A brief experimental study using real-world data sets shows that queries in ISEQL can be evaluated efficiently.

The remainder of the paper is organized as follows. In the next section we cover related work and in Section 3 we give an overview of the proposed framework and a formal description of the low-level and medium-level events. Section 4 defines the medium-level events with interval relations and introduces the operators of our extended relational algebra, while Section 5 illustrates how to use these operators to detect high-level events. In Section 6, we describe in detail the overall framework developed for event detection. This is followed by an experimental evaluation investigating the accuracy and run time of our approach. Finally, Section 8 concludes the paper.

Related Work

There is a substantial body of work on searching for specifically defined patterns of activities in a video surveillance context (Hongeng, 2001) using stochastic automata: Vaswani (2005) studies Hidden Markov Models (HMMs), Brand (1997) and Oliver (2002) use coupled HMMs, Hamid (2003) employs Dynamic Bayesian Networks (DBNs) to capture causal relationships between observations and hidden states, Albanese (2007) develops a stochastic automaton-based language, and Cuntoor (2008) presents an HMM-based algorithm.