YARN

MapReduce NextGen aka YARN aka MRv2

The new architecture introduced in hadoop-0.23, divides the two major functions of the JobTracker: resource management and job life-cycle management into separate components.

The new ResourceManager manages the global assignment of compute resources to applications and the per-application ApplicationMaster manages the application’s scheduling and coordination.

An application is either a single job in the sense of classic MapReduce jobs or a DAG of such jobs.

The ResourceManager and per-machine NodeManager daemon, which manages the user processes on that machine, form the computation fabric.

The per-application ApplicationMaster is, in effect, a framework specific library and is tasked with negotiating resources from the ResourceManager and working with the NodeManager(s) to execute and monitor the tasks.

More details are available in the Architecture document.

Documentation Index

YARN

• YARN Architecture

• Capacity Scheduler

• Fair Scheduler

• ResourceManager Restart

• ResourceManager HA

• Web Application Proxy

• YARN Timeline Server

• Writing YARN Applications

• YARN Commands

• Scheduler Load Simulator

• NodeManager Restart

• DockerContainerExecutor

• Using CGroups

• Secure Containers

• Registry

YARN REST APIs

• Introduction

• Resource Manager

• Node Manager