This chapter covers the essentials of big data analytics ecosystems primarily from the business and technology context. It delivers insight into key concepts and terminology that define the essence of big data and the promise it holds to deliver sophisticated business insights. The various characteristics that distinguish big data datasets are articulated. It also describes the conceptual and logical reference architecture to manage a huge volume of data generated by various data sources of an enterprise. It also covers drivers, opportunities, and benefits of big data analytics implementation applicable to the real world.
TopIntroduction
In Information Age, we are overwhelmed with data, ways to store, process, analyze, interpret, consume and act upon the data. The term Big Data is quite vague and ill defined. The word “Big” is too generic and the question is how “Big” is considered as “Big” and how “Small” is small (Smith, 2013) is relative to time, space and circumstance. The size of “Big Data” is always evolving and the meaning of Big Data Volume would lie between Terabyte (TB) and Zettabyte (ZB) range. The concept of big data is the explosion of data from the Internet, cloud, data center, mobile, Internet of things, sensors and domains that possess and process huge datasets. Cisco claimed that humans have entered the ZB era in 2015 (Cisco, 2017).
Based on social media statistics 2018, the face book claimed that, there are over 300 million photos uploaded to Facebook every day (Nowak & Spiller, 2017). On an average 300 hours of videos are uploaded every minute on You Tube (YouTube, 2017). Approximately, 42 billion texts are sent and 1.6 billion photos shared through Whatsapp daily (Stout, 2018). Since 2005, business investment in hardware, software, talent, and services has increased as much as 50 percent, to $4 trillion (Rijmenam, 2018).
In 2005, Roger Mougalas from O’Reilly Media coined the term Big Data for the first time. It refers to a large set of data that is almost impossible to manage and process using traditional business intelligence tools. During the same year, Yahoo created Hadoop. This was built on top of Google’s MapReduce. Its goal was to index the entire World Wide Web (Rijmenam, 2018).
In 2009, the Indian government decides to take an iris scan, fingerprint and photograph of all of its 1.2 billion inhabitants. All this data is stored in the largest biometric database in the world (Chandra, 2018).
In 2010, at Technomy conference, Eric Schmidt stated, “There were 5 Exabyte’s of information created by the entire world between the dawn of civilization and 2003. Now that same amount is created every two days.” (Schmidt, 2010).
In 2011, McKinsey released a report on Big Data which claimed that, the next frontier for innovation, competition, and productivity, states that in 2018 the USA alone will face a shortage of 140.000 – 190.000 data scientist as well as 1.5 million data managers (Manyika, 2011).
Another detailed review was contributed by Visualizing.org (Hewlett Packard Enterprise, 2017) in Big Data. It is focused on the time line of how to implement Big Data Analytics. Its historical description is mainly determined by events related to the Big Data push by many internet and IT companies such as Google, YouTube, Yahoo, Facebook, Twitter and Apple. It emphasized the significant impact of Hadoop in the history of Big Data Analytics.
In the past few years, there has been a massive increase in Big Data startups, trying to deal with Big Data and helping organizations to understand Big Data and more and more companies are slowly adopting and moving towards Big Data.
Figure 1 shows the history of Big Data and its eco system.
The data sources and their formats are continuous to grow in variety and complexity. Few list of sources includes the public web, social media, mobile applications, federal, state and local records and databases, commercial databases that aggregate individual data from a spectrum of commercial transactions and public records, geospatial data, surveys and traditional offline documents scanned by optical character recognition into electronic form. The advent of the more Internet enabled devices and sensors expands the capacity to collect data from physical entities, including sensors and radio-frequency identification (RFID) chips. Personal location data can come from GPS chips, cell-tower triangulation of mobile devices, mapping of wireless networks, and in-person payments (Manyika, 2011). The big challenge is, how do we consume those data sources and transform them into actionable information Big Data describes a data management strategy that integrates many new types of data and data management alongside traditional data.