Abstract
Entity resolution (also known as duplicate elimination) is an important part of the data cleaning process, especially in data integration and warehousing, where data are gathered from distributed and inconsistent sources. Learnable string similarity measures are an active area of research in the entity resolution problem. Our proposed framework builds upon our earlier work on entity resolution, in which fuzzy rules and membership functions are defined by the user. Here, we exploit neuro-fuzzy modeling for the first time to produce a unique adaptive framework for entity resolution, which automatically learns and adapts to the specific notion of similarity at a meta-level. This framework encompasses many of the previous work on trainable and domain-specific similarity measures. Employing fuzzy inference, it removes the repetitive task of hard-coding a program based on a schema, which is usually required in previous approaches. In addition, our extensible framework is very flexible for the end user. Hence, it can be utilized in the production of an intelligent tool to increase the quality and accuracy of data.
Key Terms in this Chapter
Data Warehouse: A data warehouse is a database designed for the business intelligence requirements and managerial decision making of an organization. The data warehouse integrates data from the various operational systems and is typically loaded from these systems at regular intervals. It contains historical information that enables the analysis of business performance over time. The data are subject oriented, integrated, time variant, and nonvolatile.
Mamdani Method of Inference: Mamdani’s fuzzy inference method is the most commonly seen fuzzy methodology. It was proposed in 1975 by Ebrahim Mamdani as an attempt to control a steam engine and boiler combination. Mamdani-type inference expects the output membership functions to be fuzzy sets. After the aggregation process, there is a fuzzy set for each output variable that needs defuzzification. It is possible, and in many cases much more efficient, to use a single spike as the output membership function rather than a distributed fuzzy set. This type of output is sometimes known as a singleton output membership function, and it can be thought of as a “predefuzzified” fuzzy set. It enhances the efficiency of the defuzzification process because it greatly simplifies the computation required by the more general Mamdani method, which finds the centroid of a two-dimensional function. Rather than integrating across the two-dimensional function to find the centroid, you use the weighted average of a few data points. Sugeno-type systems support this type of model.
Machine Learning: Machine learning is an area of artificial intelligence concerned with the development of techniques that allow computers to learn. Learning is the ability of the machine to improve its performance based on previous results.
OLTP (Online Transaction Processing): OLTP involves operational systems for collecting and managing the base data in an organization specified by transactions, such as sales order processing, inventory, accounts payable, and so forth. It usually offers little or no analytical capabilities.
OLAP (Online Analytical Processing): OLAP involves systems for the retrieval and analysis of data to reveal business trends and statistics not directly visible in the data directly retrieved from a database. It provides multidimensional, summarized views of business data and is used for reporting, analysis, modeling and planning for optimizing the business.
Data Cleaning: Data cleaning is the process of improving the quality of the data by modifying their form or content, for example, removing or correcting erroneous data values, filling in missing values, and so forth.
Sugeno Method of Inference: Introduced in 1985, it is similar to the Mamdani method in many respects. The first two parts of the fuzzy inference process, fuzzifying the inputs and applying the fuzzy operator, are exactly the same. The main difference between Mamdani and Sugeno is that the Sugeno output membership functions are either linear or constant.