Performance evaluation plays a crucial role in the development and improvement of search systems in general and context-based systems in particular. In order to evaluate search systems, test collections are needed. These test collections typically involve a corpus of documents, a set of queries and a series of relevance assessments. In traditional approaches users or hired evaluators provide manual assessments of relevance. However this is difficult and expensive, and does not scale with the complexity and heterogeneity of available digital information. This chapter proposes a semantic evaluation framework that takes advantages of topic ontologies and semantic similarity data derived from these ontologies. The structure and content of the Open Directory Project topic ontology is used to derive semantic relations among a massive number of topics and to implement classical and ad hoc retrieval performance evaluation metrics. In addition, this chapter describes an incremental method for context-based retrieval, which is based on the notions of topic descriptors and topic discriminators. The incremental context-based retrieval method is used to illustrate the application of the proposed semantic evaluation framework. Finally, the chapter discusses the advantages of applying the proposed framework.
TopIntroduction
Contextual search is the process of seeking information related to a user’s thematic context (Budzik et al., 2001; Maguitman et al., 2005a; Kraft et al., 2006; Ramirez & Brena, 2006). As is the case with conventional information retrieval (IR), evaluating the performance of contextual search requires experimenting with test collections. Relevance judgments of semantic relationships between topics and Web pages are crucial at the moment of building these test collections. However, human assessments of relevance are very hard to come by, especially when dealing with large and heterogeneous corpora. As a consequence, collecting relevance judgments is especially challenging and is one of the major bottlenecks in the development of thematic retrieval evaluation frameworks.
Automating the process of obtaining relevance judgments consistent with human-based assessments could greatly facilitate the implementation of robust evaluation tools for IR in general and contextual search in particular. Editor-driven topic ontologies such as ODP (Open Directory Project - http://www.dmoz.org) have enabled the design of automatic evaluation methodologies (Beitzel et al., 2005; Haveliwala et al., 2002; Menczer 2004). In these methodologies a document is assumed to be relevant to a specific topic if the document is classified under that topic. This gives rise to a large collection of relevance judgments for diverse topics. Since documents are classified into topics by human editors, these judgments can be taken as a gold standard and therefore are extremely valuable for evaluation purposes. In addition, the topic ontology hierarchical structure gives rise to semantic similarity measures between Web pages assigned to different but related topics (Haveliwala et al., 2002; Menczer 2004).
Inferring suitable semantic similarity measures based on the structure of topic ontologies is essential for the implementation of a fair evaluation framework based on ontologies such as the ODP. However, most existing approaches focus only on the hierarchical component of the ODP and fail to capture many semantic relationships induced by the ontology’s non-hierarchical components (symbolic and related links). As a result, according to these approaches, the semantic similarity between Web pages in topics that belong to different top-level categories is zero even if the topics are clearly related.
In light of this limitation Maguitman et al. (2005b) proposed an information theoretic measure of semantic similarity that can be applied to objects stored in the nodes of arbitrary graphs, in particular topical ontologies that combine hierarchical and nonhierarchical components such as Yahoo!, ODP and their derivatives. Therefore, it can be usefully exploited to derive semantic relationships between massive numbers of topics as well as between those Web pages assigned to these topics, giving way to the design of more precise automatic evaluation methodologies than those that are based only on the hierarchical component of these ontologies. This makes it natural to implement a framework for assessing the performance of contextual search systems.