Mining the Internet for Concepts

Abstract

The Internet has made available a big number of information services, such as file sharing, electronic mail, online chat, telephony and file transfer. However, services that provide effective access to Web pages, such as Google, are the ones that most contributed to the popularization and success of the World Wide Web and the Internet. Pages published at the World Wide Web belong to many different topic areas, such as music, fishing, travel, etc. Some organizations have tried to organize pages in a predefined classification, and have manually built large directories of topics (e.g. Dmoz or the Yahoo! directory). But given the huge size and the dynamic nature of the Web, keeping track of pages and their topic manually is a daunting task. There is also the problem of agreeing on a standard classification, and this has proved to be a formidable problem, as different individuals and organizations tend to classify things differently. Another option is to rely on automatic tools that mine the Web for “topics” or “concepts” related to online documents. This approach is indeed more scalable than the manual one. However, automatically classifying documents in topics is a major research challenge. This is because the document keywords alone seem to be insufficient to directly convey the meaning of the document to an autonomous system. In some cases, the main difficulty is due to the ambiguity of the terms encountered in the document. Even if the ambiguity problems were solved there is still no guarantee that the vocabulary used to describe the document will match that used by the autonomous system to guide its search. Central to automatic approaches is the notion of “semantic context”, which loosely means the subject or topic where a task like searching is embedded. Of course, we need a way to computationally represent this notion of context, and one possibility is to see context as a collection of interrelated terms in the sense that they appear together in a number of related pages (Ramirez & Brena, 2006). For instance, the word “Java” appears together with “roasted” when talking about coffee, but appears more frequently with “code” when talking about a programming language. Semantic contexts allow performing searches on the Web at the concept level, rather than at the more basic keyword level. In this chapter we present recent advances in automated approaches in web concept mining, emphasizing our own work about mining the Web for semantic contexts.