A Predictive Map Task Scheduler for Optimizing Data Locality in MapReduce Clusters

2. Architecture Of Mapreduce

MapReduce is a programming model for large-scale data- intensive distributed data processing. It divides the execution in two phases: map and reduce. In the map phase, amounts of map tasks process data blocks independently. After all map tasks are finished, the reduce phase begins. The intermediate results of map tasks are shuffled, sorted, and processed in parallel with one or more reduce tasks.

A user submits a job comprising of a map function and a reduce function which are subsequently transformed into map and reduce tasks scheduled on slots hosted by participating nodes in the cluster. HDFS loads data partitions into fixed equal-size splits, and distributes splits across cluster nodes. Each split is assigned a map task.