Integrated Data Mining and Business Intelligence

Background

Today, the collection and analysis of data is integral to the strategic performance of an organization. When performance does not meet targeted expectations organizations must be in a position to analyse performance data to gain insight on how relevant strategies can be improved. They need to find out why they are having problems, what the cause is and the optimal approach for improvement.

Due to the volumes and complexity of data collected across organizations today manual approaches to analysis are often not effective. The data may be spread across the organisation in a diversity of systems and formats. Analytical systems need to be able to integrate this diversity and provide a comprehensive view of the business. Automated analytic systems are required to help sift through large volumes of data to find interesting patterns that would not be possible manually. Data mining techniques provide, well-proven, techniques to help automate analysis. Data Mining helps transform data into actionable information and provide the insight required to improve strategy (Han, J. & Kamber, M., 2001). What is commonly known as Data Mining is a part of a small process called “Knowledge Discovery in Databases”? According to Piatetsky-Shapiro and Frawley (Piatetsky-Shapiro & Frawley, 1991), “Knowledge Discovery in Databases is the non-trivial process of identifying valid, novel, potentially useful, and ultimately understandable patterns in data”.

Knowledge Discovery is a cyclic process. The process starts with the selection of key data from a data warehouse, data mart or the amalgamation of data from other sources (Edelstein H. 1999). When large data bases are available it is often necessary to sample the data to get smaller manageable portions of data. Usually the data requires manual pre-processing or transforming to some degree prior to analysis. Pre-processing may involve tasks like cleaning up the data, remove noisy, missing data, or grouping data. Transformation usually involves the creation of new derived attributes. e.g. Trends or Moving Averages but also may involve filtering, ordering, editing, and normalisation. Data visualization is also important at this point to gain a basic understanding of the data.Data mining is the next part of this process, according to Bradley et al., (Bradley, Fayyad, & Mangasarian, 1998): “Data mining is the step in the KDD process concerned with the algorithmic means by which patterns or models (structures) are enumerated from the data under acceptable computational efficiency limitations.” Data Mining can be used in a number of ways:

• •

  Exploration: The aim is to gain new insight on your business and discover new patterns in your data. We may discover a small segment of customers that are not satisfied and discover their locality is not covered well by service technicians.

• •

  Confirmation: You may have a hunch that a pattern exists and need to test for its existence.

• •

  Prediction: The aim is to build a model that can predict the likelihood of business events. The model reads in new records and generates a score for that event. For examples, predicting if a customer will churn next month, or respond to marketing campaign, or predict when a system is going to fail.

• •

  What if Analysis: Once you have a predictive model it can be used to explore different scenarios by entering hypothetical data.

Data Mining can be used to help solve the following kinds of problems: