Images in biomedical articles are often referenced for clinical decision support, educational purposes, and medical research. Authors-marked annotations such as text labels and symbols overlaid on these images are used to highlight regions of interest which are then referenced in the caption text or figure citations in the articles. Detecting and recognizing such symbols is valuable for improving biomedical information retrieval. In this research, image processing and computational intelligence methods are integrated for object segmentation and discrimination and applied to the problem of detecting arrows on these images. Evolving Artificial Neural Networks (EANNs) and Evolving Artificial Neural Network Ensembles (EANNEs) computational intelligence-based algorithms are developed to recognize overlays, specifically arrows, in medical images. For these discrimination techniques, EANNs use particle swarm optimization and genetic algorithm for artificial neural network (ANN) training, and EANNEs utilize the number of ANNs generated in an ensemble and negative correlation learning for neural network training based on averaging and Linear Vector Quantization (LVQ) winner-take-all approaches. Experiments performed on medical images from the imageCLEFmed’08 data set, yielded area under the receiver operating characteristic curve and precision/recall results as high as 0.988 and 0.928/0.973, respectively, using the EANNEs method with the winner-take-all approach.
Top1. Introduction
Authors of biomedical publications use images to illustrate medical concepts and highlight special cases. These images often convey essential information and can be very valuable for improved clinical decision support (CDS) and education. Biomedical information retrieval has, so far, been largely text-based and limited mostly to bibliographic information. To be of greater value, it is desirable to retrieve images from biomedical publications. However, they need to be first annotated with respect to their usefulness for CDS to help determine relevance to a clinical query or to queries for special cases important in educational settings (Demner-Fushman, 2007, 2008, 2009).
Image retrieval can be achieved using the following methods: (i) traditional text-based approaches that index figure captions, (ii) image retrieval approaches that index the visual content of the images, and (iii) an intelligent combination of the above. To enhance text-based retrieval, content-based image retrieval (CBIR) has been explored to retrieve information from images in the biomedical field (Demner-Fushman, 2007). However, the approaches have not taken advantage of specific image regions of interest (ROIs) highlighted by the author using overlaid symbols, such as arrows and other text labels, and identifying them in the caption text. Further, it has been shown that whole image retrieval without attention to specific regions of interest marked by annotations, such as arrows (Figure 1), is not as promising as retrieval of text, primarily due to “semantic gap” introduced by less relevant image regions (Deserno, 2009). It is commonly understood in the field that low level features such as color, texture, and shape used in CBIR are insufficient to represent medical concepts or meaningful diagnostic information in the images effectively unless they can be applied to the key image regions such as those identified by the author, as in the case of images from biomedical articles. To improve the relevance quality of conventional retrieval approaches, we have proposed an approach using hybrid (text and image) features (Antani, 2008; You, 2009, 2010). Information retrieval techniques are used to identify key textual features in the title, abstract, figure caption, and figure citation (“mention”) in the article. Structured vocabularies, such as the National Library of Medicine’s Unified Medical Language System (UMLS®) are used as well to identify the biomedical concepts in these (Demner-Fushman, 2009; You, 2009). Unlike conventional CBIR schemes that extract features from the entire image, our approach uses a combination of features: those computed from specific image region of interests (ROIs) in addition to the ones obtained from the entire image. The ROIs are detected by localizing and recognizing image annotations such as arrows overlaid by authors. Annotations and ROIs in retrieved images can be identified by the annotation recognizer and then used to re-rank the results of the recognizer.
Figure 1. Medical image with arrows (a) Arrow example (adapted from Caskey, 1999) (b) Non-arrow example (adapted from Fraser, 1999)
There are some techniques that have been implemented to find arrows in previous research. Sparce pixel vectorization has been explored to detect arrowheads (Dov & Liu, 1999a, 1999b). In addition, arrow sign identification has been investigated for robot navigation using a camera-based method (Park, 2008). Compared to the existing approaches, the arrow symbols seen in the medical images experimental data set used in this research have a more complex shape. As shown in Figure 1(a), arrows in these medical images do not necessarily have to be straight (see arrow 3, arrow 4 in Figure 1(a)) and the shape of the arrows can be significantly different (see arrow 2 in Figure 1(a)) as well. Furthermore, the example image in Figure 1(b) shows objects such as characters and symbols which can be of similar size to arrows, providing potential false arrow detections. Therefore, a general and robust arrow detection algorithm is needed for discrimination from other medical image artifacts.