Article Preview
Top1. Introduction
With an increasing demand for Big Data and data science skills, companies are facing a shortage of job candidates with the necessary talent. As early as 2011 Gartner projected that by 2018 there would be a shortage of 140,000 – 190,000 data scientists, along with a shortage of 1.5 million managers that could understand and interpret the analysis generated by the data scientists (Manyika et al., 2011). More recent studies have shown that employers continue to identify an increasing need for Big Data and data science skills (Wixom et al. 2014). This demand in the job market has led to an emphasis in academia on the development of courses, certificates, (and more recently degree programs) in data science and analytics, but mainly at the graduate level (Chen, Chiang, & Storey, 2012).
While the focus in data science has been on statistics, machine learning, and predictive analytics, leading to the data scientist’s role being described as the sexiest job of the 21st century, 50% - 90% of the work in data science is the data acquisition and data wrangling needed to prepare data for analytics (Balboni, Finch, Reese, & Shockley, 2013; Davenport, 2014; Rattenbury, Hellerstein, Heer, & Kandel, 2015). Additionally, non-technical skills often sought when hiring data science teams include an insatiable curiosity about data, the ability to formulate data questions, and domain business knowledge. In a recent study of analytics and data science programs at the undergraduate level, Aasheim, Williams, Rutner, & Gardiner, (2015) found that data mining and analytics/modeling were covered in 100% of such programs. Of the four major areas reviewed, the area least frequently covered was Big Data. However, Big Data is the fuel that powers data science.
Despite the growth in data science programs, particularly at the graduate level, the demand for employees who understand Big Data, data science, and business will exceed the number of students graduating from these programs. One alternative is to build services that can scale up while abstracting away some of the complexities of Big Data. This was the original approach that Google took with distributed computing in developing MapReduce; allowing programmers to utilize distributed computing without each programmer needing to address the underlying complexities and system failures inherent in distributed computing.
The research questions we examine in this paper are: (1) In a semester course, can students learn a framework for scoping data questions and apply it to their own projects? (2) Through a series of lab exercises can undergraduate students successfully apply Big Data services as part of a cloud-based stack to answer their own student-generated data questions using a current social media dataset?
To answer these questions, we analyzed the results of student teams from an undergraduate course focused on Big Data. This course teaches students to work with semi-structured data, formulate business questions to be answered, wrangle the data, profile the data using Hadoop on Microsoft’s Azure cloud platform, and visualize the results using an iterative approach.
The rest of this paper is organized as follows. In Section 2 we review the literature on Big Data with an emphasis on data wrangling and Big Data as a service. Section 3 discusses the methodology, structure, and tools used in the class and how they fit into the overall structure of the course, Section 4 presents our results, and section 5 concludes the paper.