Knowledge Combination vs. Meta-Learning

Introduction

Research in intelligent information systems investigates the possibilities of enhancing their over-all performance, particularly their prediction accuracy and time complexity. One such discipline, data mining (DM), processes usually very large databases in a profound and robust way (Fayyad et al., 1996). DM points to the overall process of determining a useful knowledge from databases, that is, extracting high-level knowledge from low-level data in the context of large databases. This article discusses two newer directions in this field, namely knowledge combination and meta-learning (Vilalta & Drissi, 2002).

There exist approaches to combine various paradigms into one robust (hybrid, multistrategy) system which utilizes the advantages of each subsystem and tries to eliminate their drawbacks. There is a general belief that integrating results obtained from multiple lower-level decision-making systems, each usually (but not required) based on a different paradigm, produce better performance. Such multi-level knowledge-based systems are usually referred to as knowledge integration systems. One subset of these systems is called knowledge combination (Fan et al., 1996). We focus on a common topology of the knowledge combination strategy with base learners and base classifiers (Bruha, 2004).

Meta-learning investigates how learning systems may improve their performance through experience in order to become flexible. Its goal is to search dynamically for the best learning strategy. We define the fundamental characteristics of the meta-learning such as bias, and hypothesis space.

Section 2 surveys the various directions in algorithms and topologies utilized in knowledge combination and meta-learning. Section 3 represents the main focus of this article: description of knowledge combination techniques, meta-learning, and a particular application including the corresponding flow charts. The last section presents the future trends in these topics.

Background

So far, commonly utilized decision-making systems have been exploiting a single technique, strategy, or topology. Consequently, their accuracy and overall performance have not been so high (Pratt & Thrun, 1997). New data mining (DM) systems utilize results obtained from several lower-level systems, each usually (but not required) based on different paradigm, or combine or refine them within a dynamic process. Thus, such a multi-strategy (hybrid) system consists of two or more individual ‘agents’ that interchange information and cooperate together.

It should be noted that there are in fact two fundamental approaches for combining the information from multi-data tasks:

• 1.

  In data combination, the data sets are merged into a single set before the actual knowledge acquisition.

• 2.

  In knowledge (theory) combination, or sensor fusion, several agents (base classifiers, sensors) process each input data set separately, and the induced models (knowledge bases) are then combined at the higher-level.

When we look at the issue of the multi-strategy systems from the other side, we come to the meta-learning. Generally speaking, meta-learning investigates the way the learning systems can increase their performance and efficiency over experience.

The base learners, the ones with a simple inductive paradigm, such as algorithms inducing decision trees or decision sets of rules, or neural nets, generate a hypothesis (concept description) by applying a fixed bias that is implanted in the knowledge base of the learner. The performance usually increases by larger training sets and losing the restrictions on the hypotheses (concept descriptions).

Using other words, a meta-learner searches dynamically for the best learning strategy and consequently, its performance is flexible. There are a few strategies of the meta-learning, however, various researches recognize it in various ways so that one cannot specify exactly which strategy belongs to meta-learning and which not (Vilalta & Drissi, 2002). Also, there is no sharp boundary between knowledge combination and meta-learning; some researches on machine learning (ML) and DM claim that the first is the subset of the latter, some not. Therefore, this article introduces the most common sights to this issue.