Integration of data sources refers to the task of developing a common schema as well as data transformation solutions for a number of data sources with related content. The large number and size of modern data sources make manual approaches at integration increasingly impractical. Data mining can help to partially or fully automate the data integration process.
TopBackground
Many fields of business and research show a tremendous need to integrate data from different sources. The process of data source integration has two major components.
Schema matching refers to the task of identifying related fields across two or more databases (Rahm & Bernstein, 2001). Complications arise at several levels, for example
- •
Source databases can be organized by using several different models, such as the relational model, the object-oriented model, or semistructured models (e.g., XML).
- •
Information stored in a single table in one relational database can be stored in two or more tables in another. This problem is common when source databases show different levels of normalization and also occurs in nonrelational sources.
- •
A single field in one database, such as Name, could correspond to multiple fields, such as First Name and Last Name, in another.
Data transformation (sometimes called instance matching) is a second step in which data in matching fields must be translated into a common format. Frequent reasons for mismatched data include data format (such as 1.6.2004 vs. 6/1/2004), numeric precision (3.5kg vs. 3.51kg), abbreviations (Corp. vs. Corporation), or linguistic differences (e.g., using different synonyms for the same concept across databases).
Today’s databases are large both in the number of records stored and in the number of fields (dimensions) for each datum object. Database integration or migration projects often deal with hundreds of tables and thousands of fields (Dasu, Johnson, Muthukrishnan, & Shkapenyuk, 2002), with some tables having 100 or more fields and/or hundreds of thousands of rows. Methods of improving the efficiency of integration projects, which still rely mostly on manual work (Kang & Naughton, 2003), are critical for the success of this important task.
TopMain Thrust
In this article, I explore the application of data-mining methods to the integration of data sources. Although data transformation tasks can sometimes be performed through data mining, such techniques are most useful in the context of schema matching. Therefore, the following discussion focuses on the use of data mining in schema matching, mentioning data transformation where appropriate.