Content-Based Image Retrieval for Advancing Medical Diagnostics, Treatment and Education

Introduction

Informatics and computer sciences play a key role in developing new technologies for advancing healthcare and clinical practices. Technology for healthcare and disease investigation is a highly active field of ongoing research which is frequently reviewed in the scientific literature, e.g. by Haux (1989, 2002a, 2002b, 2006, 2010) and others (Hasman, 1996; Kulikowski, 2002), and reflects the rapid advance in computer technology and performance. In medical informatics, we refer to “information logistics” when we aim at providing “the right information, at the right time, at the right place” (Reichertz, 1977, 2006). Several milestones of information logistics have already been achieved and reported in the technical literature (Haux, 2006, 2010). With respect to medical images, however, retrieval from Picture Archiving and Communication Systems (PACS) is still based on alphanumeric annotations, such as the diagnosis text, or simply the name of the patient, date of acquisition, or some study meta-information.

Content-Based Image Retrieval (CBIR) technology, on the other hand, exploits the visual content in image data. The promise of CBIR benefit to the medical community has been discussed for well over a decade. Almost 15 years ago, Tagare et al. reported on the impact expected from accessing medical image archives and mining image data by content rather than textual description (Tagare, 1997), and, in the ensuing years, CBIR in medicine has become a topic of considerable research (Deserno, 2009; Long, 2009). It has been proposed for the management of increasingly large biomedical image collections as well as to aid clinical medicine, research, and education (Antani, 2008; Müller, 2004). CBIR may be viewed as a set of methods that (1) index images based on the characteristics of their visual content, and (2) retrieve images by similarity to such characteristics, as expressed in queries submitted to the CBIR system. These characteristics, also referred to collectively as the “signature” of an image, may include intensity, color, texture, shape, size, location, or a combination of these. Sketching a cartoon, selecting an example image, or a combination of both methods, is typically used to form the query. The retrieved results are usually rank-ordered by some criteria; however, other methods, such as clustering of similar images, have been used to organize the results as well.

Practical application of CBIR depends on many different techniques and technologies, usually applied at multiple processing stages, both for the indexing as well as the retrieval workflows. These techniques may include: image segmentation and feature extraction; feature indexing and database storage of the feature indices; image similarity computation; pattern recognition and machine learning; image compression and networking for image storage and transmission; and Internet technologies (such as JavaScript, PHP, AJAX, Applet/Servlet). Human factors and usability considerations may also play a role in the system design and implementation although, as we shall discuss, they appear to be under-emphasized. More recently, natural language processing has also been included, in attempts to exploit text descriptions of image content and the availability of standardized vocabularies (Névéol, 2009). It is through careful selection of appropriate methods from these fields that a successful CBIR application can be developed.

The technical literature regularly reports on experimental implementations of CBIR algorithms and prototype systems, yet the application of CBIR technology for either biomedical research or routine clinical use appears to be very limited. While there is widespread enthusiasm for CBIR in the engineering research community, the incorporation of this technology to solve practical medical problems is a goal yet to be realized. Possible obstacles to the use of CBIR in medicine include:

• •

  The lack of productive collaborations between medical and engineering experts, which is strongly related to usability and performance characteristics of CBIR systems;

• •

  The lack of effective representation of medical content by low-level mathematical features;

• •

  The lack of thorough evaluation of CBIR system performance and its benefit to health care; and

• •

  The absence of appropriate tools for medical experts to experiment with CBIR applications, which is again related to usability and performance attributes of CBIR systems.