Automatic Detection of Arrow Annotation Overlays in Biomedical Images

Automatic Detection of Arrow Annotation Overlays in Biomedical Images

Beibei Cheng (Missouri University of Science and Technology, USA), R. Joe Stanley (Missouri University of Science and Technology, USA), Soumya De (Missouri University of Science and Technology, USA), Sameer Antani (U. S. National Library of Medicine, USA) and George R. Thoma (U. S. National Library of Medicine, USA)
DOI: 10.4018/jhisi.2011100102
OnDemand PDF Download:
No Current Special Offers


Images in biomedical articles are often referenced for clinical decision support, educational purposes, and medical research. Authors-marked annotations such as text labels and symbols overlaid on these images are used to highlight regions of interest which are then referenced in the caption text or figure citations in the articles. Detecting and recognizing such symbols is valuable for improving biomedical information retrieval. In this research, image processing and computational intelligence methods are integrated for object segmentation and discrimination and applied to the problem of detecting arrows on these images. Evolving Artificial Neural Networks (EANNs) and Evolving Artificial Neural Network Ensembles (EANNEs) computational intelligence-based algorithms are developed to recognize overlays, specifically arrows, in medical images. For these discrimination techniques, EANNs use particle swarm optimization and genetic algorithm for artificial neural network (ANN) training, and EANNEs utilize the number of ANNs generated in an ensemble and negative correlation learning for neural network training based on averaging and Linear Vector Quantization (LVQ) winner-take-all approaches. Experiments performed on medical images from the imageCLEFmed’08 data set, yielded area under the receiver operating characteristic curve and precision/recall results as high as 0.988 and 0.928/0.973, respectively, using the EANNEs method with the winner-take-all approach.
Article Preview

1. Introduction

Authors of biomedical publications use images to illustrate medical concepts and highlight special cases. These images often convey essential information and can be very valuable for improved clinical decision support (CDS) and education. Biomedical information retrieval has, so far, been largely text-based and limited mostly to bibliographic information. To be of greater value, it is desirable to retrieve images from biomedical publications. However, they need to be first annotated with respect to their usefulness for CDS to help determine relevance to a clinical query or to queries for special cases important in educational settings (Demner-Fushman, 2007, 2008, 2009).

Image retrieval can be achieved using the following methods: (i) traditional text-based approaches that index figure captions, (ii) image retrieval approaches that index the visual content of the images, and (iii) an intelligent combination of the above. To enhance text-based retrieval, content-based image retrieval (CBIR) has been explored to retrieve information from images in the biomedical field (Demner-Fushman, 2007). However, the approaches have not taken advantage of specific image regions of interest (ROIs) highlighted by the author using overlaid symbols, such as arrows and other text labels, and identifying them in the caption text. Further, it has been shown that whole image retrieval without attention to specific regions of interest marked by annotations, such as arrows (Figure 1), is not as promising as retrieval of text, primarily due to “semantic gap” introduced by less relevant image regions (Deserno, 2009). It is commonly understood in the field that low level features such as color, texture, and shape used in CBIR are insufficient to represent medical concepts or meaningful diagnostic information in the images effectively unless they can be applied to the key image regions such as those identified by the author, as in the case of images from biomedical articles. To improve the relevance quality of conventional retrieval approaches, we have proposed an approach using hybrid (text and image) features (Antani, 2008; You, 2009, 2010). Information retrieval techniques are used to identify key textual features in the title, abstract, figure caption, and figure citation (“mention”) in the article. Structured vocabularies, such as the National Library of Medicine’s Unified Medical Language System (UMLS®) are used as well to identify the biomedical concepts in these (Demner-Fushman, 2009; You, 2009). Unlike conventional CBIR schemes that extract features from the entire image, our approach uses a combination of features: those computed from specific image region of interests (ROIs) in addition to the ones obtained from the entire image. The ROIs are detected by localizing and recognizing image annotations such as arrows overlaid by authors. Annotations and ROIs in retrieved images can be identified by the annotation recognizer and then used to re-rank the results of the recognizer.

Figure 1.

Medical image with arrows (a) Arrow example (adapted from Caskey, 1999) (b) Non-arrow example (adapted from Fraser, 1999)


Complete Article List

Search this Journal:
Volume 17: 4 Issues (2022): Forthcoming, Available for Pre-Order
Volume 16: 4 Issues (2021): 3 Released, 1 Forthcoming
Volume 15: 4 Issues (2020): 3 Released, 1 Forthcoming
Volume 14: 4 Issues (2019)
Volume 13: 4 Issues (2018)
Volume 12: 4 Issues (2017)
Volume 11: 4 Issues (2016)
Volume 10: 4 Issues (2015)
Volume 9: 4 Issues (2014)
Volume 8: 4 Issues (2013)
Volume 7: 4 Issues (2012)
Volume 6: 4 Issues (2011)
Volume 5: 4 Issues (2010)
Volume 4: 4 Issues (2009)
Volume 3: 4 Issues (2008)
Volume 2: 4 Issues (2007)
Volume 1: 4 Issues (2006)
View Complete Journal Contents Listing