A Hybrid Approach to Big Data Systems Development

Introduction

Big data is an emerging area which has only recently caught the attention of researchers. Data has always played a critical role in decision making, and these analyses typically use structured (or well-behaved) data. In the digital age, however, there is an abundance of untapped unstructured data that needs to be mined and analyzed for useful information. This data is generated via social networking, sensors, mobiles, apps and many smart devices. Much of this data contains important information that is still unexplored. The volume, speed and variety with which this data is generated is prompting both the public and private sectors to think about new ways of managing and analyzing data--big data analytics is an outgrowth of this thinking. Literature has typically focused on social networks and data mining but has not merged the areas to provide meaningful insights. Big data combines many disciplines such as analytics, statistics, database, sociology, etc., in order to provide useful information. Many isolated big data applications have been reported in literature but there is no model that provides guidance in developing these applications. This chapter proposes a hybrid approach to developing big data applications, as it combines both structure and unstructured data in its model. This chapter uses the ethical lens theory proposed by William May in developing the hybrid model, and also describes several current applications of big data.