Whenever decision makers find out that they want to know more about how the business works and progresses, or why customers do what they do, then data miners are summoned, and business intelligence is to be built or altered. Data mining aims at retrieving valid, interesting, explicable connection between key factors for either operative reporting or supporting strategic planning. While data mining discovers static connections between factors, business intelligence visualizes relevant data for decision makers in order to make them identify fast changes and analyze precisely business states. In this chapter, the authors give a short introduction for data oriented decision support systems with data mining and business intelligence in it. While these techniques are widely used in business processes, there are much more bad practices than good ones. We try to make an attempt to demystify and clear the myths about these technologies, and determine who should and how (not) to use them.
TopIntroduction
Data-driven decision support systems (DSS) are the state-of-the-art technology for an information era company to analyze current states, to discover changes in customer behaviors, or to get a picture about present and future business. DSS consist of operative databases (current state of business), data warehouses (historic business data) for aggregations, some analytics (including data cleaning and customer anonymization), and visualization platforms (See Figure 1).
Figure 1. General overview of data driven decision support systems
Data mining processes deployed on the technical infrastructure of an enterprise is called analytics. Its purpose is either to detect some discrepancies in the data that may indicate frauds, changes in the business or the business environment, or to group customers, products, and so forth, using unique signatures in order to treat them properly. Data mining techniques are able to discover relevant features in business data, and determine latent semantics between these data, which allows for data mining algorithms to give extrapolations and predictions for future changes of these data. Once these latent semantics are discovered, a business analyst can analyze the proper features, and see whether a situation is in place or not.
Business intelligence is a process in which decision makers use interactive diagrams on time series data to retrieve information. Visualization helps analysts understand the large amount of information better as human visual processing are faster, more comprehensive, and more precise than any other human communication forms. Simple visualization techniques can only show fix states without letting know what it is behind the numbers. Business intelligence is essential to discover sudden changes, and to react faster for such changes.
Data mining and business intelligence involves a certain human factor as well. It does not only observe changes in the data, but the knowledge workers involved in the observation process also influence its output. Data mining and business intelligence trigger a learning cycle: first data miners learn, then their algorithms learn, then data analysts learn, then reporting systems learn, and finally decision makers learn. That is, the success of a data mining or a business intelligence project largely depends on proper human interactions and intuitions: on the ones who observe trends, select and refine algorithms or choose data representations adequately, and on the ones who derive further implications and conclusions according to the findings. Data mining and business intelligence algorithms themselves neither understand business, nor make implications, but discover statistically relevant correlations. Since the human factor is an essential part in this process, even a well-founded project can end up in a disaster, if not properly controlled or verified.
This chapter presents the fundamental steps towards building up business intelligence based on data mining, and in each step we highlight the most typical human mistakes and misbelieves we found in the over 40 data mining projects we audited or attended as a consultant in the last 10 years. These projects cover various sectors including health care, bank and insurance businesses, telecommunications, and public services.
In this chapter, we use the term data mining for structured data only (i.e., data stored in relational schemas or as a stream of records, e.g., time series) without lack of generality. Literature distinguishes different mining procedures based on media types (e.g., text mining, machine vision which is image or video mining), and/or on different applications (e.g., recommendation engines, sentiment analysis, video analytics) because tools, contexts, problems, and therefore applicable tools and algorithms are different, however the principles to extract knowledge remain still the same.
TopHow Data Mining Works
In the ‘90s, knowledge discovery in databases (KDD) was defined as a “non–trivial (automated) process of identifying valid, novel, potentially useful, and ultimately understandable patterns in (large) data” (Frawley, Piatetsky-Shapiro, & Matheus, 1992), and the “analysis step”(SeeFigure 2) of this process was called data mining. Nowadays, we rather use the term data mining in a bit different sense: data mining is an automated process or an organization of large data sets in order to extract novel, valid and potentially useful behavioral information. That is, all discovered correlations: