Big Data: Challenges and Solutions

Introduction

This chapter updates the most important characteristic about how computing framework must be configured and intelligently administers to fulfill the main aspects related to security required by the applications of the Big Data. The place where the huge quantity of data is stored, extracted and processed is called Big Data. Big Data is the area with vast potential to handle datasets of the size which is beyond the capability of generally used software tools to capture, manage, and timely analyze that amount of data. The amount of data to be analyzed is estimated to be twice for every two years. The various sources of this unstructured data are medical records, scientific applications, sensors, social media, video and image archives, surveillance, business transactions, Internet search indexing, and system logs. As the number of connecting devices of Big Data like Internet of things increases attention towards Big Data also increased at unexpected levels. In addition, it is very common to acquire on-demand supplementary computing control and storage to carry out exhaustive data-parallel processing from public cloud providers. Thus, privacy and security issues can be potentially increased by the variety, volume and wide area deployment of the system infrastructure to support Big Data applications. As Big Data expands with the help of public clouds, traditional security solutions tailored to private computing infrastructures, confined to a well-defined security perimeter, such as firewalls and demilitarized zones (DMZs) are no more effective. Using Big Data, security functions are required to work over the heterogeneous composition of diverse hardware, operating systems, and network domains. A clearly defined security boundary like firewalls and demilitarized zones (DMZs), conventional security solutions adapted to private computing infrastructures are no more effective for Big Data as it expands with the help of public clouds.

Using Big Data, security functions are required to work over the heterogeneous composition of diverse hardware, operating systems, and network domain. In this complicated computing environment, Big Data secure services upon the heterogeneous infrastructure are deployed efficiently by taking the abstraction capability of Software-Defined Networking (SDN). An abstraction concept of SDN separates the control (higher) plane from the underlying system infrastructure being controlled and supervised. Segregating network's control logic from the underneath physical routers and switches that forward traffic makes the system administrators to write high-level control programs that specify the behavior of an entire network, in contrast to conventional networks, whereby administrators (if allowed to do it by the device manufacturers) must codify functionality in terms of low-level device configuration. The intelligent management of secure functions simplifies the concepts of implementation of security rules; system (re)configuration; and system evolution using SDN. The main problem of a centralized SDN solution can be reduced by using a chain of controllers and/or through the usage of more number of controllers to control at least the most important functions of a system.

The National Institute of Standards and Technology (NIST) launched a framework with a set of voluntary guidelines to help organizations make their communications and computing operations safer (NIST, 2014). This could be accomplished by verifying the system infrastructure in view of security against threats, risk assessment, and ability to respond and recover from attacks.