Computer vision can be used to increase the interactivity of existing and new camera-based applications. It can be used to build novel interaction methods and user interfaces. The computing and sensing needs of this kind of applications require a careful balance between quality and performance, a practical trade-off. This chapter shows the importance of using all the available resources to hide application latency and maximize computational throughput. The experience gained during the developing of interactive applications is utilized to characterize the constraints imposed by the mobile environment, discussing the most important design goals: high performance and low power consumption. In addition, this chapter discusses the use of heterogeneous computing via asymmetric multiprocessing to improve the throughput and energy efficiency of interactive vision-based applications.
In the twilight of Moore’s Law, mainstream computers from ’desktops’ to ’smartphones’ are being permanently transformed into heterogeneous supercomputer clusters. Henceforth, a single compute-intensive application will need to harness different kinds of cores, in immense numbers, to get its job done. The free lunch is over. Now welcome to the hardware jungle. – Herb Sutter (2005)
TopTo solve the problems they face, mobile vision-based applications become more complex, leading to tight requirements in order to efficiently address the computations they involve. Although this is also applicable in many other fields, computer vision algorithms are particularly constrained to the processing capabilities of the hardware platforms. In this context, there is a need to maximize the computational performance of such applications by adapting them to the particularities of mobile devices.
The first issue to encounter when optimizing or porting an interactive application to a mobile platform is how to define such computational performance. The speed of a system can be characterized by two terms; latency and throughput. Throughput is defined as the amount of work done per unit time. Latency is defined as the time between the start of a process and its completion. Although interrelated, a system can be designed to optimize one of both parameters, affecting the other (Grochowski et al. 2004). For example, pipelining an algorithm could increase its throughput, but actually increase the end-to-end latency.
The extensive amount of data processed by vision-based applications implies that the system implementation has a high throughput requirement, since many times it should be able to compute several millions of pixel in less than a second. However, interactive applications and user interfaces are in practice real-time systems that require a response in a limited amount of time. In this context, the implementation must assure a latency low enough to meet the requirements of interactivity. In practice, the designer needs to carefully balance both parameters in a practical trade-off.