Studying Natural Interaction in Multimodal, Multi-Surface, Multiuser Scenarios

Background

Gestural interaction is becoming pervasive. It can be found in tablets and smartphones, who offer their users touch based interfaces, supporting direct manipulation and semaphoric gestures (Quek et al., 2002). Microsoft Kinect and other entertainment systems support deictic and semaphoric gestures also. While people interact naturally with each other through gestures, gesture dictionaries are still required for HCI. This has been acknowledged in several works that tried to understand how people interact with computers through gestures (Kurdyukova, Redlin, & André, 2012; Miki, Miyajima, Nishino, Kitaoka, & Takeda, 2008; Wobbrock, Morris, & Wilson, 2009; Yin & Davis, 2010).

While some gestures have become standard for performing actions (e.g. pinch for zooming), there is still a need to characterize the way people perform general actions in a computing environment (Dang, Straub, & André, 2009; Epps, Lichman, & Wu, 2006; Neca & Duarte, 2011). These studies show that people not only present variability in the gestures they make for each command, but also in how they make it (e.g. by using different hand posture). This impacts the interaction design of applications that want to make use of gestures, but also the way gesture recognizers need to perform. Two of the major problems identified are: (1) people perform the same gesture for different actions; and (2) people find it very difficult to come up with gestures for actions that can not be addressed through direct manipulation (e.g. deleting an object in an interactive space without a recycle bin to drop the object into).