The recognition of human behaviour from sensor observations is an important area of research in smart homes and ambient intelligence. In this chapter, the authors introduce the idea of spatio-temporal footprints, which are local patterns in space and time that should be similar across repeated occurrences of the same behaviour. They discuss the spatial and temporal mapping requirements of these footprints, together with how they may be used. As possible formalisms for implementing spatio-temporal footprints, the authors discuss and evaluate probability theory, fuzzy sets, and the Dempster-Shafer theory.
TopIntroduction
A common task that an ambient intelligence system could be required to perform is recognising human behaviour from observations in the environment; this can be useful for a variety of applications from monitoring the activities of elderly patients to identifying appropriate lighting and heating conditions (Cook, 2006; Mozer, 2005). The observations on which such recognition is based can range from direct observations made by video cameras to indirect observations detected by sensors. Although video cameras give a more complete picture, and hence might lend themselves more easily to recognising behaviours (with a consequent increase in the amount of computational processing required), it is often behaviour recognition based on sensors that is the preferred option, since the latter is less obtrusive and therefore more easily accepted in applications such as smart homes.
There is a significant body of research on behaviour recognition based on sensor data, which ranges from logic-based approaches to probabilistic machine learning approaches (Augusto & Nugent, 2004; Chua, Marsland, & Guesgen, 2009; Duong, Bui, Phung, & Venkatesh, 2005; Gopalratnam & Cook, 2004; Rivera-Illingworth, Callaghan, & Hagras, 2007; Tapia, Intille, & Larson, 2004). Although the reported successes are promising, it has become clear that all approaches fall short of being perfect. Due to the limited information that is in the sensor data, noise, and the inherently complexity of human behaviours, it is often impossible to determine the correct behaviour from the sensor data alone, in particular if behaviours are overlapping or are being executed by more than one person.
Several researchers have realised that additional information can be useful to boost the behaviour recognition process (Aztiria, Augusto, Izaguirre, & Cook, 2008; Jakkula & Cook, 2008; Tavenard, Salah, & Pauwels, 2007). In this article, we focus on how spatio-temporal information, enriched with context information, can be used for this purpose. When a particular activity occurs, like preparing breakfast, it leaves a ‘footprint’ in space-time, i.e., a particular pattern of sensor observations in some set of locations over some period of time. The activity starts at some specific time and in some specific location, goes on for a specific duration in some specific area, and terminates at a specific time at some specific location. Since footprints differ from behaviour to behaviour—but often relatively little between different instances of the same behaviour—we can use these to inform the behaviour recognition process: if something is happening at 07:00 in the kitchen, it is more likely to be preparing breakfast than taking a shower. We can also use them to detect abnormal behaviour: if the inhabitant of the smart home uses the shower at 03:00 (when usually this is not a footprint that is seen), then this can be interpreted as abnormal.
Figure 1 shows an example of a possible set of footprints over three days, with a linear time axis that repeats each day, and a single space axis that could identify rooms, or similar (this is discussed in more detail in the next section). It can be seen that some behaviours repeat more-or-less identically over the three days, while others only occur once. The challenge with such representation of behaviour as footprints is to identify and recognise the various behaviours that are represented.
Figure 1.
An example of space-time with footprints