Abstract
Large datasets pertaining to many scientific fields and everyday activities are becoming available at an increasing rate. Processing, analyzing, and understanding the information that they offer poses significant technical challenges. There are many efforts dedicated to the development of big data exploration, analysis, and visualization applications that will improve the value of the information extracted from these datasets. An analysis of the state-of-the-art in these applications is presented here along with open research challenges that have not yet been tackled sufficiently. Also, specific domains where big data applications are needed are presented, and unique challenges are identified.
TopIntroduction
This chapter presents the concept of Big Data, discusses their unique characteristics and the way their definition has evolved in time. Next, key domains where their analysis provides additional knowledge and supports decision making are presented. Then, the chapter focuses on two Big Data research fields that try to address the needs of a wider audience, people that can greatly benefit from the available datasets but without access to the needed infrastructure. This chapter presents Big Data exploration and visualization applications that necessitate the development of methods and techniques that can make datasets accessible, and the current state of the research along with open research challenges. The objective of this chapter is to provide the reader with a deep understanding of the Big Data era, emphasize their unique characteristics and explain how these contribute to their importance and potential. In addition, the chapter offers to the reader a clear understanding of the current research state, the areas that can be further explored and the expected next steps for the research of the field.
There are many different definitions for the term Big Data (De Mauro, 2015; De Mauro, 2016; Ward, 2013), depending on the time they were written and the field that they are referring to, but they all agree that there are large, complex and unprocessed datasets, that cannot be processed by traditional application but can offer knowledge and value if properly analyzed. Initially, there was a controversy regarding the volume of the dataset and what should be considered large or difficult to process.
This was mainly due to the fact that research has shown (Hilbert, 2011) that the application-specific capacity of the machines to compute information per capita has roughly doubled every 14 months, whereas the world's storage capacity per capita required roughly 40 months to double during the last decades.
The exponential growth of the data production, the diversity of the data sources, along with the improvement of the computational capabilities of the hardware made the quantification of the term insignificant and added multiple dimensions to the problem. To this end, a dataset is now characterized as Big Data when it complies with the seven Vs rule (Ali-ud-din Khan, 2014; Hilbert, 2016).
Key Terms in this Chapter
Data Visualization: Processing datasets and presenting the information though interactive visualizations that support the understanding of the information.
Data Analysis: Systematic processing of large datasets, often in real or near-real time, and their presentation in ways that can support the decision-making process.
Data Engineer: A software engineer responsible to prepare the needed infrastructure for big data analysis.
Data Exploration: Performing complex queries over large datasets aiming to identify patterns and extract the knowledge within.
Data Scientists: Analysts responsible to apply complex algorithms and techniques in order to extract the value from the datasets.
Big Data: There are large, complex, and unprocessed datasets that cannot be processed by traditional application but can offer knowledge and value if properly analyzed.