This chapter is an introduction to multi-cluster based anomaly detection analysis. Various anomalies present different behaviors in wireless networks. Not all anomalies are known to networks. Unsupervised algorithms are desirable to automatically characterize the nature of traffic behavior and detect anomalies from normal behaviors. Essentially all anomaly detection systems first learn a model of the normal patterns in training data set, and then determine the anomaly score of a given testing data point based on the deviations from the learned patterns. The initial step of learning a good model is the most crucial part in anomaly detection. Multi-cluster based analysis are valuable because they can obtain the insights of human behaviors and learn similar patterns in temporal traffic data. The anomaly threshold can be determined by quantitative analysis based on the trained model. A novel quantitative “Donut” algorithm of anomaly detection on the basis of model log-likelihood is proposed in this chapter.
TopBackground
The initial step of learning a good model is the most crucial part in anomaly detection. In general, an incorrect choice of model can lead to poor anomaly detection results. For example, a linear model may not work well if the underlying pattern is generated from multiple clusters. In such cases, the testing data can be mistakenly detected as anomaly because the poor fit to the learned linear model, which lead to high false alarm rates. Effective anomaly detection systems based on machine learning algorithms are hence desirable to automatically extract useful information in terms of abnormal characteristics of the systems and entities from such noisy, high-dimensional data and provide useful application-specific insights.
Machine learning algorithms include supervised learning and unsupervised learning (Jain, Murty, & Flynn, 1999; Theodoridis & Koutroumbas, 2006) . Most anomaly detection systems employ supervised algorithms based on training data, however, the training data are typically expensive to generate (Leung & Leckie, 2005). Moreover, these supervised detection techniques have difficulty in detecting new types of anomaly.
In the absence of unlabeled data, unsupervised anomaly detection techniques can be promising because they are capable of detecting previously unseen anomalies. From unsupervised learning, we are able to learn particular patterns in a way that reflects the statistical structure of the overall system. Unsupervised learning has been widely used in history. Hebb (1949) linked statistical methods to experiments. Hinton and Sejnowski (1986) invented the Boltzmann machine model, which provided insights in the density estimation methods. Some early work include Horace Barlow (1992), Donald MacKay (1956), and David Marr (1970).