Need of Hadoop and Map Reduce for Processing and Managing Big Data

Introduction

Big data is the collection of datasets that are huge in size and difficult to handle by commonly used data processing tools and its applications. These datasets are unstructured and usually originated from various sources such as social media, scientific applications, social sensors, surveillance cameras, electronic health records, web documents, archives, web logs and business applications. They are larger in the size with fast data in/out. Organizations would be interested in capturing and analyzing these datasets because they can add considerable value to the decision making process. However, such processing may involve complex workloads, which move the boundaries of what are possible using traditional data management and data warehousing techniques and technologies. Further, big data must have high value and ensure trust for decision making process. These data come from diverse sources and heterogeneity is one more important property besides volume, variety, velocity, value and veracity. Data gets collected and stored at unprecedented rates. Moreover the challenge is not only to store and manage the large amount of data, but even to analyze and extract meaningful values from it. This has contributed to the problem of big data faced by the industry due to the inability of usual database systems and software tools to manage and process the big data sets within reasonable time limits. Processing of Big data can consist of various operations depending on usage like culling, classification, indexing, highlighting, searching, faceting, etc.

Two significant data management trends for processing the big data are relational DBMS products meant for analytical workloads (also called analytic RDBMSs, or ADBMSs) and the non-relational systems (sometimes called NoSQL systems) meant for processing multi-structured data. A non-relational system can be used to generate analytics from big data or to pre-process big data before consolidated into a data warehouse.