Modern image processing systems that are recording people and their interactions (lifelogging cameras, Kinect sensors, social robots, etc.) collect enormous amount of data which is being processed in time and effort consuming way, yet far from being sufficiently competent to be reliable in performance. Most computing algorithms aim at modeling people as physical objects in dynamic interactions (e.g. Oliver, Rosario, & Pentland, 2000; Heïgeas, Luciani, Thollot, & Castagné, 2010; Kaburlasos & Papakostas, 2015). Apart from identifying other people as physical objects, human perception is capable of simultaneously identifying social interaction cues as well as psychological features that are necessary for adapting to life in the society (Forsyth, 1990; Pentland, 2007; Hari, & Kujala, 2009; Perry, Stein, & Bentin, 2011; Dimitrova, 2012). To achieve better social competence the computing systems need to be able to identify the underlying processes guiding human interactions by extracting relevant features of the perceived information for the respective abstract level of analysis by being equipped with high-level sensors for capturing and integrating multilevel sensor stimulation.
The main focus of the current chapter is the ability of a computing system to operate on the social level of description of a situation by reading the relevant cues by especially designed sensors, called high-level sensors, in a way similar to people reading social cues. The possibilities for application of the proposed theoretical approach to education of people with special learning needs or style are briefly discussed.
Background
Many current approaches to social sensor design apply the analogy with pattern recognition theories or image processing methodology. Indicators of the social nature of the object – the human – are the signifiers of social interactions and social signaling – voice intonation, hand gestures, flow of speech, posture, facial expression, etc. (Kopp, & Wachsmuth, 2004; Pentland, 2007; Pan, Cebrian, Dong, Kim, & Pentland, 2010; Courgeon, Grynszpan, Buisine, & Martin, 2012). The main approach to modeling the nature of the interaction is based on collecting sets of features and classifying behavioral patterns, belonging to different social situations (e.g. Hernandez, Reyes, Escalera, & Radeva, 2010; Kim, Helal, & Cook, 2010; Courgeon et al., 2012). Sequences of behaviors are modeled via Hidden Markov Models (Pan et al., 2010), Gaussian Mixture Models (Hernandes et al., 2010), Conditional Random Field (Kim et al., 2010), Laban movement theory (Lourens, van Berkel, & Barakova, 2010), F-formations (Hung & Krose, 2011), T-patterns (Magnusson, 2000; Jonsson, 2006), convolutional neural networks (Bolaños, Garolera, M., & Radeva, 2015), interactive fuzzy lattice reasoning (Kaburlasos & Papakostas, 2015) and other machine learning approaches. Considering that, as physical objects, people are flexible in behavior and of changing appearances, it is insufficient to simply apply the developed techniques for object and pattern recognition in other domains for detecting social presence in a situation (e.g. Groh, Lehmann, Reimers, Frieß, & Schwarz, 2010). Social presence is revealed by subtle cues suggesting the attentiveness of the agent to the social configuration between the present agents, not just their physical proximity. This attentiveness is being qualified by the observer as attitude such as being polite, impatient, empathic, etc.
It turns out that current image processing systems are capable of predicting behavior in response to behavior based on modeling people as physical objects in dynamic interactions. Next step is augmenting the computational systems with the ability to extract the relevant cues and to achieve better social communication competence, similar to the kind that is underlying human interactions. This will allow developing algorithms for predicting the outcomes of social interactions – whether conflict or consensus will result – from reading behavioral patterns in different modalities – visual (facial expressions, gestures), auditory (prosody), tactile (clash), organized in unambiguous, or integrated, sets of features. The computational system will achieve the competence of predicting behavior in response to attitude – friendly, supportive, empathic, hostile, distant, etc. – in a manner similar to the way people acquire social communication competence for better adaptation to life in the society.