Abstract
Big data is a substantial amount of data sets that cannot be stored, managed, or examined using conventional tools. Today, there are billions of data resources that generate data at a very speedy rate. These data resources are available across the world. Consider Facebook as an example—it produces 500 terabytes of data approximately every day which contains photographs, videos, text messages, emoticons, and more. In general, data that exists in the real world can be classified into different formats, like structured, semi-structured, and unstructured data. Structured data referred to as quantitative data has a well-defined format with only text /numbers which can be easily stored in the relational database. For example, data is represented in an Excel sheet. Semi-structured data has a partial format which may include tags. HTML tags fall in this category and need to be processed to store in the relational database. Unstructured data is otherwise referred to as qualitative data. For example, emails and video fall into this category.
TopBackground
Industrial Big Data
In the era of digital economic globalization, intelligent decision making has attracted a lot of attention from the digital industry market. One prime technology in artificial intelligence is big data driven analysis. This enhances the productivity and helps in making wise decisions by mining the hidden knowledge and the potential ability of the Big Data (M. Ghasemaghaei & G. Cali, 2019). Many real-time large-scale data are applied to the industrial process. Mostly the real-time data are streamed from noisy environment Also among the acquired data certain data will be labelled and few may not. Such kind of substantial amount of data with various challenges within are processed and expected to produce an optimized intelligent output without compromising the time and space dimensions. Hence the Big Data processing requires extensible methods to distribute and store real-time data, to suggest and dynamically adapt with the changes made in the process to provide automatic decisions (Ritu Ratra & Preeti Gulia, 2019). Thus, the End-to End Big Data process is expected to integrate, adapt and generalize the data in all stages within to create intelligent decisions with respect to the process.
Therefore, non-traditional techniques and strategies are required to store, organize and process the big data sets. There are several big data tools available, the following are the important big data analytics tools are highly recommendedand applied in industry servicing various needs such as data collection, data cleaning, data filtering and extraction, data validation, and data storage.
Hadoop -- To collect and evaluate data.
MongoDB -- To handle data that gets updated frequently.
Talend -- To provide data incorporation and administration.
Cassandra -- To handle aggregates of data.
Spark -- To provide real-time administration while handling large volume of data in the distributed environment.
STORM -- To process high velocity data in distributed real-time computational environment.
The following context discusses the aforementioned tools in detail with working strategies, application possibilities, its merits and exceptions in order to point out how effective the tools are applied in Big Data Analytics.
Key Terms in this Chapter
Open-Source: Denoting software for which the original source code is made freely available and may be redistributed and modified.
Web Services: A web service is any piece of software that makes itself available over the internet and uses a standardized XML messaging system.
Batch Processing: Batch data processing is a method of processing large amounts of data at once.
Dataset: A collection of related sets of information that is composed of separate elements but can be manipulated as a unit by a computer.
Distributed File System (DFS): A Distributed File System (DFS) as the name suggests, is a file system that is distributed on multiple file servers or multiple locations. It allows programs to access or store isolated files as they do with the local ones, allowing programmers to access files from any network or computer. The main purpose of the Distributed File System (DFS) is to allows users of physically distributed systems to share their data and resources by using a Common File System.
Non-Relational Databases: A non-relational database stores data in a non-tabular form, and tends to be more flexible than the traditional, SQL-based, relational database structures.
Database: A database is an organized collection of structured information, or data, typically stored electronically in a computer system.
Latency: Latency is defined as the delay before a transfer of data begins following an instruction for its transfer.
Stream Processing: Stream processing is a big data technology that focuses on the real-time processing of continuous streams of data in motion.
Data Integration: Data integration involves combining data residing in different sources and providing users with a unified view of them.