Measuring the Semantic Relatedness Between Images Using Social Tags

Introduction

Relatedness measurement especially similarity between multimedia such as images and videos plays an important role in computer vision. The image similarity is a base for many multimedia related applications including image clustering (Goldberger, 2006), searching (Evgeniou, 2003; Ji, 2010), recommendation (Fan, 2009), and annotation (Gong, 2010). The relatedness problem is relevant to two aspects: images representation and relatedness measurement. The former aspect needs an appropriate model to reserve the related information of an image. The latter aspect requires an effect method to compute the relatedness accurately.

In the early stage, relatedness measurement is based on the low-level visual features such as texture (Schmid, 1997; Varma, 2005), shape (Belongie, 2002), and gradient (Dalai, 2005). These visual features are used to represent effective information of an image. Some distance metrics including Chi-Square distance (Huang, 2009), Euclidean distance (Wang, 2005), histogram intersection (Jia, 2006), and EMD distance (Rubner, 1998) is used. Overall, these methods ignore the high-level features such as semantic information which can be understood by machine and people easily. These methods are limited to the applications which need semantic level information. Recently, with the explosion of community contributed multimedia content available online, many social media repositories (e.g. Flickr¹, Youtube², and Zooomr³) allow users to upload media data and annotate content with descriptive keywords which are called social tags. We take Flickr, one of the most popular and earliest photo sharing sites, as an example to study the relatedness measurement between images. Flickr provides an open platform for users to publish their personal images freely. The principal purpose of tagging is to make images better accessible to the public. The success of Flickr proves that users are willing to participate in this semantic context through manual annotations (Wu, 2012; Cai, 2010). Flickr uses a promising approach for manual metadata generation named “social tagging”, which requires all the users in the social network label the web resources with their own keywords and share with others. In this paper, we focus on measuring semantic relatedness between images since:

• 1.

  Semantic relatedness follows the cognitive mechanism of people. In (Van den Broek, 2010), the author suggests that the association relation is the basic mechanism of brain. When people know a concept such as “hospital”, she/he may index the related concept such as “doctor” for appropriate understanding the original concept. Since the goal of relatedness measurement is to facility related applications such as searching and recommendation, the proposed method should follow user’s cognitive mechanism.

• 2.

  Semantic relatedness can be used to organize images based on their associations. In recent literatures, such as Linked Open Data (LOD) (Bizer, 2009), Semantic Link Network (SLN) (Zhuge, 2009; Zhuge, 2011; Luo, 2011), the resources are managed by their semantic relations. The proposed semantic relatedness measures can be used to build semantic links between resources especially images, which can be easily applied in real applications.

The major contributions of this paper are summarized as follows:

• 1.

  A framework to measure semantic relatedness between Flickr images using tags is proposed. Firstly, the co-occurrence measures are used to compute the relatedness of tags between two images. Secondly, we transform the tags relatedness integration to the assignment in bipartite graph problem, which can find an appropriate matching to the semantic relatedness of images. Finally, a decline factor considering the position information of tags is used in the proposed framework, which reduces the noise and redundancy in the social tags.

• 2.

  A real data set including 1000 images from Flickr with ten classes is used in our experiments. Two evaluation methods including clustering and retrieval are performed, which shows the proposed method can measure the semantic relatedness between Flickr images accurately and robustly.

• 3.

  We extend the relatedness measures between concepts to the level of images. Since the association relation is the basic mechanism of brain. The proposed relatedness measurement can facility related applications such as searching and recommendation.