Optimization of the Knowledge Discovery Process in very Large Databases

Abstract

Current database technology involves processing a large volume of data in order to discover new knowledge. The high volume of data makes the discovery process computationally expensive. In addition, real-world databases tend to be incomplete, redundant and inconsistent which could lead to discovery of redundant and inconsistent knowledge. We propose use of domain knowledge to reduce the size of the database being considered for discovery and to optimize the hypothesis (representing the pattern to be discovered) by eliminating implied, unnecessary and redundant conditions from the hypothesis. The benefits can be greater efficiency and the discovery of more meaningful, non-redundant, non-trivial and consistent rules. Experimental results are provided and analyzed.