Data Science and Distributed Intelligence

Introduction

The two terms Big Data (Stonebraker & Hong, 2012) and MapReduce (Dean & Ghemawat, 2008) have dominated the scene in the intelligent data analysis field during the last two years. They are in fact the cause and effect of the rapid growth in data observed in the digital world. The phenomenon of very large databases and very high rate streaming data has been coined recently as Big Data. The largest two databases for Amazon account for 42 terabytes of data in total, and YouTube receives at least 65,000 new videos per day. Such figures increase every day and people are literally drowning in high waves of data. Making sense out of this data has become more important than ever in the knowledge era. With the birth of learning from data streams, Mutukrishnan in his later published book (Muthukrishnan, 2005) has defined data streams as “data arriving in a high rate that challenges our computation and communication capabilities.” In fact, this definition is now more true than back then. In spite of the continuous advances in our computation and communication capabilities, the data growth has been much faster, and the problem has become even more challenging. As a natural reaction to this worsening, a number of advanced techniques for data streams have been proposed, ranging from compression paradigms (e.g., (Cuzzocrea et al., 2004a, 2004b, 2005; Cuzzocrea & Chakravarthy, 2010)), mainly inherited by previous experiences in OLAP data cube compression (e.g., (Cuzzocrea, 2005; Cuzzocrea & Serafino, 2009)) to intelligent approaches that successfully exploit the nature of such data sources, like their multidimensionality, to gain in effectiveness and efficiency during the processing phases (e.g., (Cuzzocrea, 2009)), and recent initiatives that are capable of dealing with complex characteristics of such data sources, like their uncertainty and imprecision, as dictated by modern stream applicative settings (e.g., social networks, Sensor Web, Clouds – (Cuzzocrea, 2011)).

Addressing such challenges has kept Data Mining and Machine Learning practitioners and researchers busy with exploring the possible solutions. MapReduce has come as a potentially effective solution when dealing with large datasets, by enabling the breakdown of the main process into smaller tasks. Each of these tasks could be performed either in a parallel or distributed processing mode of operation. This allows the speed-up of performing complex data processing tasks, in an attempt to catch up with high speed large volume of data generated by scientific applications (Jiang et al., 2010), such as the promising contexts of analytics over large-scale multidimensional data (e.g., (Cuzzocrea et al., 2011)) and large-scale sensor network data processing (e.g., (Yu et al., 2012)). With Big Data and MapReduce at the front of the scene, a new term describing the process of dealing with very large dataset has been coined, Data Science.

In line with this, when these kind of dataset are processed on top of a service-oriented infrastructure like the novel Cloud Computing one (Agrawal et al., 2011), the terms “Database as a Service” (DaaS) (Hacigumus et al., 2002) and “Infrastructure as a Service” (IaaS) arise, and it is become critical to understand how Data Science can be coupled with distributed, service-oriented infrastructures, with novel and promising computational metaphors. Hence, due to the inherent distributed nature of computational infrastructures like Clouds (but also Grids (Foster et al., 2001)), it is natural to view Distributed Intelligence as the most natural underlying paradigm for novel Data Science challenges.