Opportunities and Challenges of Big Data in Public Sector

Big Data Development Process

Big data system development process is similar to other systems, with the difference being the scale and nature of computing. The following steps are necessary for a project of this magnitude to succeed:

• •

  Collect

• •

  Manage/architecture

• •

  Process

• •

  Act

Each of these steps has its challenges. By some estimates, 90% of the data generated by devices is not useful. Therefore, the first step is to identify relevant data which requires functional expertise. Managing data would require data integration of structured and unstructured data. Several non-relational databases (key-value, key-document, graph) are emerging that can be used. However, the processing of this structure can be quite challenging. Typical processing is based on massive parallel processing that typically uses a HADOOP like structure. This requires creating clusters and replicating them. The task is to create the appropriate cluster to balance load. Once data is stored and defined it needs to be processed. Traditional languages like SQL are not appropriate. The challenge is to develop new languages that can help in processing. Once data is processed, the next step is to understand the output and apply it.

The next section describes opportunities in the public sector.