An Image Clustering and Feedback-Based Retrieval Framework

Introduction

Object-based image retrieval has recently become an important research issue in retrieving images on the basis of the underlying semantics of images. Within the context of object-based image retrieval, the semantic content of an image is represented by one or more of the objects present in that image, where object-based features are used to describe the user’s perceived content of that image. However, two critical issues exist in existing object-based image retrieval systems. First, most existing systems retrieve images according to a single object/segment of a user’s interest, which cannot meet the requirement of those user queries where more than one object-of-interest is involved. In addition, image segmentation, known as an extremely difficult process, may produce inaccurate segmentation resulting from over-segmentation and/or under-segmentation. Consequently, the inaccurate segments may negatively affect the image retrieval results. Secondly, most existing systems require a complex user interface which is capable of displaying all the segments (regions) of a query image so that users can choose desirable query region(s) from these segments. Such an interface can be cumbersome and confusing, mostly due to the inaccurate segmentation results by over- and under-segmentation, but also partly due to the fact that there are usually 7~8 segments/objects on average in each image.

Integrated Region Matching (IRM) (Carson, Thomas, & Belongie, 1999) has been proposed to alleviate the above two problems to some degree. IRM measures overall similarity between two images with the following two advantages. First, IRM effectively reduces the side effect of inaccurate segmentation by incorporating properties of all the regions in the images into one region matching scheme. Moreover, unlike other existing object-based image retrieval systems, IRM does not require a complex user interface to display all the segments/objects in the query image because IRM adopts an overall image-to-image similarity measure based on the similarity of two sets of objects/segments. Therefore, users only need to specify a query image without having to specify particular objects of interest through a complex user interface. However, several challenges remain in IRM, including: (1) how to efficiently index and search in a large-scale image segment/object database, and (2) how to bridge the “semantic gap” between low level object features and high level perceptions of image content consisting of a set of objects. As an unsupervised similarity measure, the original IRM ranks the retrieved images based on the overall similarity between two sets of image segments without any input of user guidance/knowledge, where the significance of each image segment is fully determined by low level object features such as region/object size. The significance score is then used to calculate the pair-wise segment similarity score between a segment of the query image and a segment from an image in the database. The overall similarity score is the sum of pair-wise segment similarity scores between two images. However, this matching scheme failed to capture the user’s preferences such as the relative importance of certain objects according to the user’s subjective perception, and a selected few regions that form the profile of the user’s search interest in a query image. In other words, the original IRM scheme does not reflect the relative importance of individual regions/objects in the query image according to the user’s own preferences. This article aims to design an object-based image clustering and retrieval framework with feedback-based integrated region matching to address the challenges aforementioned.