Concept Drift Detection in Data Stream Clustering and its Application on Weather Data

Introduction

With the advancement in hardware and software technology, the number of applications producing large volume data streams is ever increasing. These data streams become useful to the industry and society only when the valuable information contained in them is extracted. Various data stream mining algorithms are available for this purpose. Capability to process the data in a single scan, requirement of online and incremental updation of the models, adaptation to the concept changes etc. are some challenges while designing learning algorithms for data streams. Clustering is an important machine learning task applied on data streams to gain useful insights into the natural groupings in data. It also helps to track the development of various phenomena in application fields like meteorology, healthcare and astrophysics (Bhatnagar, 2009).

The main advantage of data stream mining is that it assumes the data source or the process generating the stream is not stationary. According to the changes in the environment that produces the stream, the underlying data distributions change over time. Consequently, these changes might affect the inter relationship between input and output variables (Gama, 2014) leading to ‘Concept Drift' (Widmer and Kubat, 1996; Gama, 2014). In these situations, the learned model becomes obsolete and the prediction accuracy reduces considerably. Weather data is a classic example where the concept can change over time (Widmer and Kubat, 1996). Depending on the seasons, weather prediction rules might exhibit a change, i.e., the same relation might not hold well in all the seasons.

This paper proposes a framework for the online clustering of data streams. It performs concept drift detection and cluster evolution monitoring to generate a warning on the dynamic changes taking place in the environment of the stream. Studies revealed that concept drift accompanied by clustering structure changes often imply important physical events. Recording such inter-relationships between the occurrence of concept drift and the corresponding physical events over a period of time can help the prediction of these physical phenomena by cluster analysis.

The proposed framework has components to cluster the stream online, detect concept changes and track the evolution of clusters. Before performing the clustering, the best value for the ‘number of clusters’, k is computed dynamically. This is done with the intention to identify the changes in the clustering structure for the recently arrived data. As the source of the data is highly dynamic, the clustering structure also might exhibit a corresponding change and fixing the value of k limits the ability to capture such changes in the clustering structure.

The utility of this framework is studied by applying it on weather data. Concept drifts, changes in the best value of k and the cluster evolutions are being monitored to understand their relationship with the physical weather phenomena. Nowadays most of the weather monitoring equipment produces high-speed data with a large number of variables. Weather stream produced by the Automatic Weather Station (AWS) at the Advanced Centre for Atmospheric Radar Research (ACARR) of Cochin University of Science and Technology, Kerala, India is used for this study. Data is collected at one-minute intervals and it contains the weather parameters like temperature, relative humidity, wind speed, wind direction, radiation, pressure, rainfall, etc. Monsoons are the most important weather phenomenon as far as the Indian region is concerned. Hence the framework is used to study the interrelationship between the changes in the clustering structure and the evolution of the south-west monsoon.

The paper is structured as follows. Section 2 refers to the background of this work. Section 3 details the proposed framework with subsections on each component of the framework. Section 4 explains the application of the proposed framework on weather data. Section 5 details about the experiments and results followed by ‘discussion and future work’ in section 6. The paper is concluded in section 7.