This chapter aims to develop an ‘accessory-free’ or ‘minimum accessory’ interface used for communication and computation without the requirement of any specified gadgets such as finger markers, colored gloves, wrist bands, or touch screens. The authors detect various types of gestures, by finding fingertip point locations in a dynamic changing foreground projection with varying illumination on an arbitrary background using visual segmentation by reflectance modeling as opposite to recent approaches which use IR (invisible) channel to do so. The overall performance of the system was found to be adequately fast, accurate, and reliable. The objective is to facilitate in the future, a direct graphical interaction with mobile computing devices equipped with mini projectors instead of conventional displays. The authors term this a dynamic illumination environment as the projected light is liable to change continuously both in time and space and also varies with the content displayed on colored or white surface.
Top1. Introduction
With an ever increasing role of computerized machines in society, the need for more ergonomic and faster Human Computer Interaction (HCI) systems has become an imperative. Here, we aim to develop an ‘accessory-free’ or ‘minimum accessory’ interface suitable for communication and computation without the requirement of often used gadgets such as finger markers, colored gloves, wrist bands, or touch screens. We describe here a robust method to detect various types of gestures. Our approach works by locating different salient parts, specifically fingertips, in a spatio-temporally dynamic foreground projection. This projection itself constitutes the varying illumination which the gestures have to be detected: moreover, we allow this projection to fall upon a nearly arbitrary background surface. The overall performance of the system was found to be adequate for real-time use, in terms of speed, and accurate and reliable enough in a practical setting. The long term objective is to eventually facilitate a direct graphical interaction with mobile computing devices equipped with mini projectors instead of conventional displays. It must be noted that unlike the conventional setting in which intrusions are detected as regions of major change in a ‘learned’ static background, our ‘background’ is in fact the instantaneous displayed output of the computing device, and is therefore generally liable to vary in space and time with the content displayed. Furthermore, keeping in mind the exigencies of anywhere-anytime computing, the system we propose does not require a plain white surface to be available to display upon: instead, it only requires that the surface should have at all points, a certain minimum non-specular reflectance and also be planar, even if not strictly normal to the projector-camera axis. According to most currently reported approaches, such an unconstrained problem specification would necessitate the use of an IR (invisible) channel for the finger intrusion detection. Our approach operates exclusively with visual detection and applies the principle of reflectance modeling on the scene where intrusion needs to be detected and also on intrusion which in our case is hand to achieve this. Briefly, it consists of the following two steps:
- 1.
A process we call Dynamic Background Subtraction, under varying illumination upon an arbitrary background using a reflectance modeling technique that carries out visual detection of the shape of intrusion on the front side projected background. This particular process in patented by us in India (Application No: 974/DEL/2010)
- 2.
Detecting the gestures and quantifying them: this is achieved by specially tuned light algorithms for the detection of the contour trajectory of the intruding hand through time, and tracking multiple salient points of this intrusion contour. Gestures can then be classified and subsequently quantified in terms of the extracted multi trajectory parameters such as position, velocity, acceleration, curvature, direction, etc.
A special, simplified, case of the above general approach is the demonstrated Paper Touchpad which functions as a virtual mouse for a computer, operating under conditions of stable (non-dynamic) illumination on arbitrary backgrounds, with the requirement of a single webcam and a piece of paper upon which the ‘touchpad’ is printed. This is an interactive device easy to use anywhere, anytime and employs a homographic mapping between screen and piece of paper. The paper touchpad, however, does not obviate the display.
In the end, we aim to design a robust real time system which can be embedded into a mini-projector and camera equipped mobile device that can be used without accessories anywhere a flat surface and some shade (from excessively bright light such as direct sunlight) is available. The single unit would substitute for the computer or communicator, the display, keyboard, mouse, a piano, a calculator etc.