Framework for Integration of Medical Image and Text-Based Report Retrieval to Support Radiological Diagnosis

Framework for Integration of Medical Image and Text-Based Report Retrieval to Support Radiological Diagnosis

Siddhivinayak Kulkarni (MAEER's MIT College of Engineering, India), Amit Savyanavar (MAEER's MIT College of Engineering, India), Pradnya Kulkarni (Federation University, Australia & MAEER's MIT College of Engineering, India), Andrew Stranieri (Federation University, Australia) and Vijay Ghorpade (Dr. D. Y. Patil College of Engineering and Technology, India)
Copyright: © 2018 |Pages: 37
DOI: 10.4018/978-1-5225-2829-6.ch006
OnDemand PDF Download:
List Price: $37.50


Medical images are vital part of diagnostics and patient treatment. With the advent of technology, there is a rapid increase in the number of radiological images produced every day. Attempts have been made to use a Content Based Image Retrieval (CBIR) approach for assisting in radiological diagnosis. However, this approach suffers from the semantic gap problem. Few text retrieval systems are in place for assisting the radiologist to retrieve similar past cases. However, for the least experienced radiologist it is hard to describe the unknown case using text query. Therefore, the aim of this chapter is integrating the radiological CBIR and text based reports retrieval in order to support radiological diagnosis. The proposed technique is described in three stages: a) retrieval by image similarity, b) retrieval by text, and c) fusion of image and text retrieval for better diagnosis. Number of experiments are demonstrated along with their evaluation techniques on mammogram image database.
Chapter Preview


Background and Motivation

Imaging of body parts by using different techniques such as X-ray, MRI, Ultrasound, and Computed tomography (CT) is helpful in diagnosing serious diseases such as cancer. Due to advancement in technology and growth in population, there is a vast increase in the number of radiological images. For instance about 200,000 images were generated per day in one of the hospitals in Geneva in 2011 ( Radiology images are used for many applications including diagnostics, therapy and teaching.

It has been observed that diagnostic radiologists struggle to maintain high interpretation accuracy while trying to fulfil efficiency targets (Rubin, 2000). Misdiagnosis by radiologists due to non-medical reasons are reported to be in the range of 2% to 4% (Siegle et al., 1998; Soffa et al, 2004). The misinterpretation of cancer lesions increases patients’ anxiety, health-care costs and also results in death in rare cases. According to (Muramatsu et al., 2005), radiological image interpretation consists of three tasks; perception of image findings, interpretation of these findings to render a differential diagnosis and recommendations for clinical management. The perception of image findings and their interpretation requires two types of knowledge. One is formalized domain knowledge which is present in texts and other written documentation. Second is implicit or tacit knowledge consisting of radiologists’ individual expertise, organisational practices and past cases (Montani and Bellazzi, 2002).

In the diagnostic scenario if a radiologist fails to express the unknown abnormality in text query, then the text retrieval for a similar case would naturally fail. And so, Query by image content has been more successful in retrieving similar past cases to help diagnosis (Shyu et al., 1999; Aisen et al., 2003; Xue et al., 2009; Fischer et al., 2012; Zhou et al., 2012).

The proposed system will be designed to handle following two scenarios:

  • Search for images with lesions that look similar and query text describing what the radiologist thinks is present as a clinical feature/finding.

  • Search for images with lesions that look similar and text query text describing the diagnosis the radiologist is thinking about.

According to radiologists, there could be more than one clinical features or findings that look the same on mammograms. For example a Cyst and a Mass would look very similar on a mammographic image. Also, the clinical feature/finding present could possibly be because of more than one disease. Table 1 below shows some of the examples.

Table 1.
Different diagnosis for same clinical findings in mammograms

Complete Chapter List

Search this Book: