This chapter explains common methods in evaluating model predictive power. If the goal is defined as finding the most important/risky customers, there are many different ways using the available resources. Analysts measure accuracy and look for answers. It is obvious that two different analysts would provide different models; however, what both are looking for is an adequate level of accuracy. That means that analysts have freedom while looking for models, but the final model needs to be accurate and usable for decision making. No matter what the final model is, the most important factors before the final results are confirmed are the model relevance tests. One can, for example, create several models with the same goal but using different methods or methodologies. The one with highest accuracy level is the best one. It is important to point out that models do not have to be based only on one method but can combine several methods at the same time.
Top9.1 Principle 80:20
Looking for ideal environment for prediction, aim is not to deal with individuals but to form a foundation able to deals with groups or behavior in wider perspective. In the light of any individual memory of previous events, prediction models have to deal with wide number of attributes in databases which are related to specific group or segment and are created by combining data from individuals. We can say that environment for prediction, and also churn, is really a collective „memory“ created from processed data patterns and „conclusions“ made with appliance of analytical methods. While creating and using model, analyst is trying to mimic human cognitive processes. We can use simple equation to express this:
Knowledge = Memory + Intelligence
(1)In order to discover knowledge, we need history (data, memory) and intelligence based on methods and models. While using methods with historic data, we gain knowledge which will be further used in business practice. For any model but especially more complex ones, it is always a question how efficiently we can predict future events by looking at the past ones. Researches so far shows that, especially in modern fast changing economy, historic data has to be used (we need to learn from past) however with constant re-evaluation of added value research is bringing to table for prediction of the future. We can explain process of modeling by looking into future using analytical rear-view made of databases and intelligent computational methods.
Not like traditional history analysis, analytic modelling is also based on iterative process of building models known as iterative or spiral approach. Usage of this approach, allows constant model improvement and development together with errors minimization using back propagation principle. Right after first prediction, future became past and system can learn on its errors, understanding problems in previous predictions and making every new iteration step more accurate and efficient.
If we are analyzing these kind of models and its capabilities understanding its own errors, we need to point out that human help is still dominant factor for efficiency because although those systems are more and more advanced every day, they are still no table to completely auto adapt and autocorrect on their own. Historic data are not useful only for prediction but also when we want to analyze market segments or specific clients.
Historic data can help us in situations when we are looking for answers on questions like:
- •
Why sales trend is decreasing?
- •
What causes decreasing trend?
- •
What to do to improve sales?
- •
What to do to decrease cost level?
- •
If we decrease price on specific product, can we improve its sales volume?
- •
Which customers are most likely to end relationship and move for competitor's offer?
If we take a look at those questions, we can find that human cognitive process also look for historic data while looking for conclusions. In churn analysis we are looking for same methodology. It is obvious that using analytical methods we can find answers for many questions, for process to be efficient those question needs to be structured in very clear way. Problem needs to be defined very precisely while question needs to be structured in a way which will give clear and undoubtful guidelines to analyst who will build final model. We can illustrate this with simple question quiz: what is 25=? Possible solutions are illustrated in Table 1.
Table 1. 25=? |
5*5 |
5+20 |
10+15 |
27-2 |
(3+2)*5 |
0.1+24.9 |
... |