Article Preview
Top1. Introduction
A common task that an ambient intelligence system could be required to perform is recognising human behaviour from observations in the environment; this can be useful for a variety of applications from monitoring the activities of elderly patients to identifying appropriate lighting and heating conditions (Cook, 2006; Mozer, 2005). The observations on which such recognition is based can range from direct observations made by video cameras to indirect observations detected by sensors. Although video cameras give a more complete picture, and hence might lend themselves more easily to recognising behaviours (with a consequent increase in the amount of computational processing required), it is often behaviour recognition based on sensors that is the preferred option, since the latter is less obtrusive and therefore more easily accepted in applications such as smart homes.
There is a significant body of research on behaviour recognition based on sensor data, which ranges from logic-based approaches to probabilistic machine learning approaches (Augusto & Nugent, 2004; Chua et al., 2009; Duong et al., 2005; Gopalratnam & Cook, 2004; Rivera-Illingworth et al., 2007; Tapia et al., 2004). Although the reported successes are promising, it has become clear that all approaches fall short of being perfect. Due to the limited information that is in the sensor data, noise, and the inherently complexity of human behaviours, it is often impossible to determine the correct behaviour from the sensor data alone, in particular if behaviours are overlapping or are being executed by more than one person.
Several researchers have realised that additional information can be useful to boost the behaviour recognition process (Aztiria et al., 2008; Jakkula & Cook, 2008; Tavenard et al., 2007). In this article, we focus on how spatio-temporal information, enriched with context information, can be used for this purpose. When a particular activity occurs, like preparing breakfast, it leaves a ‘footprint’ in space-time, i.e., a particular pattern of sensor observations in some set of locations over some period of time. The activity starts at some specific time and in some specific location, goes on for a specific duration in some specific area, and terminates at a specific time at some specific location. Since footprints differ from behaviour to behaviour—but often relatively little between different instances of the same behaviour—we can use these to inform the behaviour recognition process: if something is happening at 07:00 in the kitchen, it is more likely to be preparing breakfast than taking a shower. We can also use them to detect abnormal behaviour: if the inhabitant of the smart home uses the shower at 03:00 (when usually this is not a footprint that is seen), then this can be interpreted as abnormal.
Figure 1 shows an example of a possible set of footprints over three days, with a linear time axis that repeats each day, and a single space axis that could identify rooms, or similar (this is discussed in more detail in the next section). It can be seen that some behaviours repeat more-or-less identically over the three days, while others only occur once. The challenge with such representation of behaviour as footprints is to identify and recognise the various behaviours that are represented.
Figure 1. An example of space-time with footprints.
The rest of this article investigates the role of spatio-temporal footprints in more detail. We start with a discussion of what space-time means in the context of smart homes, arguing that there is more than one space-time (or more precisely, representation of space-time). We then look at how behaviours leave footprints in space-time and explore invariants in these footprints, with the goal of classifying different forms of invariants. Finally we will look into how the footprints are distributed in space-time and what influences this distribution.