Real-Time Vehicle Traffic Prediction in Apache Spark Using Ensemble Learning for Deep Neural Networks

Introduction

With the dawn of the era of Big Data upon us, there has been a huge surge in the volume of available data and it is anticipated to increase several folds in the coming years. All-embracing research is being carried out to put this data into effective use. The growth in big data and big data technologies have opened up interesting prospects that were once unforeseen by researchers and businesses. Research on how to take advantage of the available data with Artificial Intelligence is achieving enthralling and striking results. One of the areas which needs the consideration of this is vehicular traffic analysis which has emerging demands in today’s world. As stated by the World Urbanization Prospects by United Nations, the urban population of the world has increased swiftly from 751 million in 1950 to 4.2 billion in 2018 which accounts for 55% of the world’s population, a proportion that is likely to surge to 68% by 2050 (Anonymous, 2018). With such rapid urbanization, vehicular traffic on roads is set to intensify drastically leading to congestion and accidents. This issue of increasing vehicle traffic needs to be addressed by exploiting big data technologies and machine learning to offer efficient solutions that offer convenience for commuters. With the easy and wide accessibility of massive amounts of data along with the advancement in big data analytics, it has been made possible to bring progress in traffic analysis.

Though the development in technology has already on track creating a great impact on the field of transportation and traffic management, by addressing difficulties and providing advanced, effective and intelligent solutions, it is still a challenging problem to carry out real time traffic analysis. This is because of the intricacies in analysing and learning from enormous volumes of real time streaming data and offer accurate predictions which need more consideration. This can be expedited by leveraging Machine Learning and big data techniques such as stream processing to develop more advantageous and desirable solutions. In order to process data streams at a faster rate, we need high computing capacity. Big data frameworks like Apache Spark, Hadoop and Kafka, with their capability of processing a multitude of data, have resolved the major issue of processing and storing continuously flowing data stream. With the progress of various big data frameworks that process large volumes of data in a distributed computing environment, it is now convenient and easy to manage, process, analyse and store data for real time analytics. These big data technologies along with the remarkable growth in the field of artificial intelligence, have made it plausible to develop advanced and efficient data stream processing systems that can curb the problem of this ever-increasing vehicular traffic congestion around the world by real time traffic analysis.

The proposed work focuses on analysing the traffic data and thus predicting traffic flow by utilizing deep Learning techniques and an efficient big data framework based on Hadoop MapReduce called Apache Spark that effectively handles unstructured, real time, streaming data. The deep learning model used to predict traffic is an ensemble of many deep neural networks. The significance of the proposed work is the implementation of an end to end traffic prediction framework by integrating Apache Kafka, Apache Spark and MongoDB for streaming, processing and storing of traffic data along with the use of the deep neural network ensemble learning model for efficiently performing real time traffic analysis and forecast. The model predicts the traffic flow in a lane at different times of a particular day or for the upcoming days based on the historical as well as real-time data by using an ensemble of neural networks which yields a better accuracy in prediction than an individual neural network.