Automated Evaluation of Interest Point Detectors

Introduction

Interest points are pixel coordinates highlighting a location that contains a simple localised feature. These often serve as a means for tracking a scene's camera motion, or an object's motion over time, and are therefore highly relevant to augmented reality applications (Gauglitz, Höllerer, & Turk, 2011; Gil, Mozos, Ballesta, & Reinoso, 2010). In Schmid's classic work (2000), the performance of interest point detectors was evaluated by manipulating a camera and a backlit projector. Such a manual approach to image capture and homography estimation has since been used in a number of related studies (Gauglitz, Höllerer, & Turk, 2011; Gil, Mozos, Ballesta, & Reinoso, 2010; Guillaume Galés, 2010; Olague & Trujillo, 2011) and typically relies on marked-up images or predictable, physical transformations of real scenes to measure the performance of a detector. While this strategy implies a higher chance of inaccuracies and smaller data sets, it has remained largely unchanged, although its application has expanded over the years. Our paper therefore aims to revisit Schmid's work and recreate the same experimental conditions in a much more powerful and versatile virtual 3D environment. This way we can utilise complex and precise transformations that would have been time consuming or impossible to perform in a real world environment. The position of interest points on 3D models can thus be quickly and accurately determined so that their repeatability—and by extension the performance of the detector that found these points—can be properly measured, as illustrated in Figure 2.

Figure 2.

Illustration of 3D model being rotated from 0° to 50°. 3D correspondences are determined via inverse projection and rotated by 50°, so that each point in can be mapped (and vice versa, not shown). Repeatability can now be measured easily by t testing for points in close neighbourhood.

Background

The most common method of evaluation of interest points in a scene is based on repeatability. By measuring the repeatability of interest points in two slightly different scenes and , we can determine how well a given detector performs (Schmid, Mohr, & Bauckhage, 2000). Consider that we compare detected interest points and from and . The distance of any two given interest points can be determined by using a homography of an interest point in two different images of a scene. The repeatability of interest points is gauged by its locality within a radial threshold . Any interest points that appear in both scenes (denoted and ) and are within the threshold distance are considered repeated (as shown in Figure 1). This determines a set of “repeated” points as:

Figure 1.

Schmid repeatability: the points and are the projections of a 3D point onto images and . A detected point is repeated if is detected in the neighbourhood of