Apache Hadoop YARN (Yet Another Resource Negotiator) is the cluster resource management technology in Hadoop version 2. The YARN provides multiple accesses for batch and real-time processing across multiple clusters and has the benefit over utilization of cluster resources during dynamic allocation. The chapter shows the YARN architecture, schedulers, resource manager phases, YARN applications, commands, and timeline server. The architecture of YARN splits the task into resource management and job scheduling. This is managed by Resource Manager and Node Manager. The chapter addresses the Timeline Server, which stores and retrieves the present and past information in a generic way.
TopBackground
Hadoop being the foundation of the big data era, there is processing model difference between hadoop 1 and hadoop 2. Hadoop 1 includes the progress with HDFS for storage and the processing by batch oriented MapReduce jobs. The version 1 is potential for distributed processing but not much suitable for interactive analysis along with memory intensive algorithms. Hence hadoop 2 includes new version of HDFS federation and resource manager YARN. HDFS federation combines the measure of scalability and reliability to hadoop, YARN supports and implements a flexible execution engine with high end processing models. It further separates processing and resource management of Mapreduce in Hadoop 1. Additionally, it is responsible for administering workloads with security controls and to maintain multi tenant environment amid high availability features.
Figure 1.
Hadoop 1 => Hadoop 2 Comparison
TopYarn Architecture
YARN (Yet Another Resource Negotiator) is split up with two major tasks i.e. resource management and job scheduling to act on a global environment and as per application. An application can either refer to a job or a DAG of jobs. In Hadoop 1, inflexible slots are configured on nodes which gets underutilized when more map or reduce tasks are running and also can’t share resources with non map reduce applications running on Hadoop cluster like Impala, Giraph etc. A host in Hadoop refers to a node and cluster is the connection of two or more nodes joined by a high speed network. Nodes may be partitioned in racks and blocks as in hardware part of hadoop infrastructure. There can be thousands of hosts in a cluster. In Hadoop, there is one master host with multiple worker hosts. Master sends jobs to the worker host. The basic overall structure of Apache Hadoop running environment including YARN is given in Figure 2.
Figure 2. Structure of Apache Hadoop v2
In Figure 2, MapReduce framework is the software layer implementing the map and reduce tasks paradigm. The HDFS Federation provides permanent, reliable and distributed storage for input and output whereas storage can include any alternative storage service from other providers like Amazon.
YARN is responsible for computational processing and memory for application executions. The data computation framework in YARN is formed by resource manager and the node manager. These two form the new generic system for managing applications in a distributed processing network environment. YARN and HDFS are independent since one provides resource and other provides storage.
TopResource Manager And Node Manager
The Resource Manager (RM) acts as eventual authority to manage resources among all jobs presently executing in the system. The Node Manager (NM) is per host agent which is responsible for containers to monitor the usage of memory, CPU, network etc. and in turn it states to the Resource Manager Scheduler. There exists an Application Master per host which is a library to execute and monitor tasks in Node Manager and also negotiates the resources from Resource Manager.