Abstract
Current databases are able to store several Tbytes of free-text documents. The main purpose of a database from the user’s viewpoint is the efficient information retrieval. In the case of textual data, information retrieval mostly concerns the selection and the ranking of documents. The selection criteria can contain elements that apply to the content or the grammar of the language. In the traditional database management systems (DBMS), text manipulation is restricted to the usual string manipulation facilities, i.e. the exact matching of substrings. Although the new SQL1999 standard enables the usage of more powerful regular expressions, this traditional approach has some major drawbacks. The traditional string-level operations are very costly for large documents as they work without task-oriented index structures. The required full-text management operations belong to text mining, an interdisciplinary field of natural language processing and data mining. As the traditional DBMS engine is inefficient for these operations, database management systems are usually extended with a special full-text search (FTS) engine module. We present here the particular solution of Oracle; there for making the full-text querying more efficient, a special engine was developed that performs the preparation of full-text queries and provides a set of language and semantic specific query operators.
TopText Mining
The subfield of document management that aims at processing, searching, and analyzing text documents is text mining. The goal of text mining is to discover the non-trivial or hidden characteristics of individual documents or document collections. Text mining is an application oriented interdisciplinary field of machine learning which exploits tools and resources from computational linguistics, natural language processing, information retrieval, and data mining.
Key Terms in this Chapter
Ranking Engine: A module within the FTS engine that ranks the documents of the result set based on their relevance to the query.
Stemmer: It is a language-dependent module that determines the stem form of a given word. The stem form is usually identical to the morphological root. It requires a language dictionary.
Fuzzy Matching: A special type of matching where the similarity of two terms are calculated as the cost of the transformation from one into the other. The most widely used cost calculation method is the edit distance method.
Thesaurus: A special repository of terms, which contains not only the words themselves but the similarity, the generalization and specialization relationships. It describes the context of a word but it does not give an explicit definition for the word.
Query Refinement Engine: A component of the FTS engine that generates new refined queries to the initial query in order to improve the efficiency of the retrieval. The refined queries can be generated using the users’ response or some typical patterns in the query history.
Indexer: It builds one or more indices for the speed up information retrieval from free text. These indices usually contain the following information: terms (words), occurrence of the terms, format attributes.
Inverted Index: An index structure where every key value (term) is associated with a list of objects identifiers (representing documents). The list contains objects that include the given key value.
Full-Text Search (FTS) Engine: A module within a database management system that supports efficient search in free texts. The main operations supported by the FTS engine are the exact matching, position-based matching, similarity-based matching, grammar-based matching and semantic-based matching.
Word-Braker: A component of the full-text engine whose function is to break the text into words and phrases.
Sectioner: A component of the FTS engine, which breaks the text into larger units called sections. The types of extracted sections are usually determined by the document type.