大数据hadoop学习之---Yarn 体系架…

最新推荐文章于 2020-11-23 16:42:27 发布

李荣强

最新推荐文章于 2020-11-23 16:42:27 发布

阅读量404

点赞数

分类专栏： hadoop

本文链接：https://blog.csdn.net/li951418089/article/details/50392778

版权

hadoop 专栏收录该内容

5 篇文章 0 订阅

订阅专栏

Apache Hadoop NextGen MapReduce (YARN)

MapReduce has undergone a complete overhaul in hadoop-0.23 and we now have, what we call, MapReduce 2.0 (MRv2) or YARN.

The fundamental idea of MRv2 is to split up the two major functionalities of the JobTracker, resource management and job scheduling/monitoring, into separate daemons. The idea is to have a global ResourceManager (RM) and per-application ApplicationMaster (AM). An application is either a single job in the classical sense of Map-Reduce jobs or a DAG of jobs.

The ResourceManager and per-node slave, the NodeManager (NM), form the data-computation framework. The ResourceManager is the ultimate authority that arbitrates resources among all the applications in the system.

The per-application ApplicationMaster is, in effect, a framework specific library and is tasked with negotiating resources from the ResourceManager and working with the NodeManager(s) to execute and monitor the tasks.

大数据hadoop学习之---Yarn 体系架构英文原版及个人翻译和理解 - passion - passion

The ResourceManager has two main components: Scheduler and ApplicationsManager.

The Scheduler is responsible for allocating resources to the various running applications subject to familiar constraints of capacities, queues etc. The Scheduler is pure scheduler in the sense that it performs no monitoring or tracking of status for the application. Also, it offers no guarantees about restarting failed tasks either due to application failure or hardware failures. The Scheduler performs its scheduling function based the resource requirements of the applications; it does so based on the abstract notion of a resource Container which incorporates elements such as memory, cpu, disk, network etc. In the first version, only memory is supported.

The Scheduler has a pluggable policy plug-in, which is responsible for partitioning the cluster resources among the various queues, applications etc. The current Map-Reduce schedulers such as the CapacityScheduler and the FairScheduler would be some examples of the plug-in.

The CapacityScheduler supports hierarchical queues to allow for more predictable sharing of cluster resources

The ApplicationsManager is responsible for accepting job-submissions, negotiating the first container for executing the application specific ApplicationMaster and provides the service for restarting the ApplicationMaster container on failure.

The NodeManager is the per-machine framework agent who is responsible for containers, monitoring their resource usage (cpu, memory, disk, network) and reporting the same to the ResourceManager/Scheduler.

The per-application ApplicationMaster has the responsibility of negotiating appropriate resource containers from the Scheduler, tracking their status and monitoring for progress.

MRV2 maintains API compatibility with previous stable release (hadoop-1.x). This means that all Map-Reduce jobs should still run unchanged on top of MRv2 with just a recompile.

个人翻译和理解:

MapReduce 到0.23版本已经终结了,我们现在有了2.0版本的mapreduce,也就是Yarn.

yarn 的基本原理就是分割JobTracker的两大功能: 资源管理与任务调度及监控,将这两个功能分割为

单独的守护进程.

基本的思想就是有一个全局的资源管理器和基于应用的appMaster(每个应用一个appmaster),

这里提及的应用就是传统意义上的map-reduce任务或者DAG任务

资源管理器节点ResourceManger以及每个slave节点即NodeManager 组成数据计算框架.

ResourceManager管理器节点对资源的丛书有完全的分配权,决定资源归属于系统中的哪个应用.

基于应用的applicationMaster 实际上就是一个特定的框架工具包,负责和ResourceManager协商获取

资源,将ResourceManager分配给的NodeManager节点的资源(slot)进行二次分配.

将任务分配给获取到的节点资源,将任务分发给这些节点,监控他们的任务执行情况.

ResourceManager 有两个主要的组件: 调度器和applicationManager

调度器负责将资源分配给多个主题相似的应用(主题相似指的是队列约束容量等等).

调度器就是一个纯粹的调度器,也就是说不负责监控和追踪应用的执行状态.

同时,不负责重启因为硬件或者应用本身出错导致的执行失败的应用.

调度器基于应用本身资源请求实现调度功能.

同样的调度器基于应用本身的资源请求

抽象出来资源容器的概念,资源容器包含了内存,cpu,磁盘网络等等元素,但是

第一个版本仅仅只支持内存.

调度器支持插件式的嵌入策略,资源调度器插件算法负责多个应用,多个队列之间,将

集群资源进行分区.

目前而言,mapreduce 调度器比如容量调度器和公平调度器是可插件式插入的例子.

资源调度器支持分级队列,对集群资源做更加可预见的分配.

applicationManager 负责接收任务的提交,协商第一个容器,

为执行任务的应用指定ApplicationMaster,同时当applicationMaster容器失败后重启

NodeManager 是资源容器的代理框架,安装在每个从节点中,负责监控当前机器的

资源使用率(cpu,内存,磁盘,网络) ,将节点中的资源使用率报告给ResourceManager 的资源调度器

基于应用的applicationMaster 负责从资源调度器的资源容器协商获取资源,追踪状态即监控进度等.

MRV2 负责维护和hadoop1 的版本兼容性,也就是说,所有用hadoop1 api写的map-reduce任务,仅仅需要重新编译就可以允许在

MRV2上.

Yarn 就是为了分离JobTracker的两大职责: 资源管理及任务调度.

其中资源管理引进了ResourceManager,统一管理资源,谁需要资源,需要通过ResourceManager申请

任务的调度有通用的接口,支持Map-Reduce,DAG,Spark,Storm等,只要符合规范,实现指定的接口按照,Yarn既定流程进行实现,就能按照

Yarn的设计初衷来走.这样就统一了企业资源分配和任务调度平台,大大提高集群的利用率.

阅读(0) | 评论(0)

喜欢推荐转载

历史上的今天

热度

在LOFTER的更多文章

Apache Hadoop NextGen MapReduce (YARN)

MapReduce has undergone a complete overhaul in hadoop-0.23 and we now have, what we call, MapReduce 2.0 (MRv2) or YARN.

', blogTag:'', blogUrl:'blog/static/2172762092015124244789', isPublished:1, istop:false, type:0, modifyTime:1424758045743, publishTime:1424758045701, permalink:'blog/static/2172762092015124244789', commentCount:0, mainCommentCount:0, recommendCount:0, bsrk:-100, publisherId:0, recomBlogHome:false, currentRecomBlog:false, attachmentsFileIds:[], vote:{}, groupInfo:{}, friendstatus:'none', followstatus:'unFollow', pubSucc:'', visitorProvince:'', visitorCity:'', visitorNewUser:false, postAddInfo:{}, mset:'000', mcon:'', srk:-100, remindgoodnightblog:false, isBlackVisitor:false, isShowYodaoAd:false, hostIntro:'JAVA软件工程师,有扎实的Java基础,熟悉JavaEE技术,对框架的底层原理熟悉,学习能力强。', hmcon:'0', selfRecomBlogCount:'0', lofter_single:'' }

{if x.visitorName==visitor.userName} ${x.visitorNickname|escape}

{else}

{/if}

{if x.moveFrom=='wap'} {elseif x.moveFrom=='iphone'} {elseif x.moveFrom=='android'} {elseif x.moveFrom=='mobile'} {/if} ${fn(x.visitorNickname,8)|escape}

{/if} {/list}

${a.selfIntro|escape}{if great260}${suplement}{/if}

{/if}

李荣强

关注

0
点赞
踩
0

收藏

觉得还不错? 一键收藏
0
评论
大数据hadoop学习之---Yarn 体系架…

<h2 style="font-size: 17px; color: rgb(51, 51, 51); font-family: Verdana, Helvetica, Arial, sans-serif; line-height: normal;">Apache Hadoop NextGen MapReduce (YARN)<a name="Apache_Hadoop_NextGen_Ma
复制链接

扫一扫