In these diagrams, the prediction space is discretised into 10 intervals (from 0 to 0.1, from 0.1 to 0.2, etc.). The examples whose probability is between 0 and 0.1 go into the first interval, the examples between 0.1 and 0.2 go into the second, etc. For each interval, the mean predicted value (in other words, the mean predicted probability) is plotted (x axis) against the fraction of positive real cases (y axis). If the model is calibrated, the points will be close to the diagonal.
Published in Chapter:
Calibration of Machine Learning Models
Antonio Bella (Universidad Politécnica de Valencia, Spain), Cèsar Ferri (Universidad Politécnica de Valencia, Spain), José Hernández-Orallo (Universidad Politécnica de Valencia, Spain), and María José Ramírez-Quintana (Universidad Politécnica de Valencia, Spain)
Copyright: © 2010
|Pages: 19
DOI: 10.4018/978-1-60566-766-9.ch006
Abstract
The evaluation of machine learning models is a crucial step before their application because it is essential to assess how well a model will behave for every single case. In many real applications, not only is it important to know the “total” or the “average” error of the model, it is also important to know how this error is distributed and how well confidence or probability estimations are made. Many current machine learning techniques are good in overall results but have a bad distribution assessment of the error. For these cases, calibration techniques have been developed as postprocessing techniques in order to improve the probability estimation or the error distribution of an existing model. This chapter presents the most common calibration techniques and calibration measures. Both classification and regression are covered, and a taxonomy of calibration techniques is established. Special attention is given to probabilistic classifier calibration.