![](https://img-blog.csdnimg.cn/20201014180756919.png?x-oss-process=image/resize,m_fixed,h_64,w_64)
hadoop-yarn
文章平均质量分 89
houzhizhen
专注大数据处理和分布式计算。
展开
-
Fair 调度器简单配置指南
当 YARN 调度器配置为 Fair 时,可以配置一个 fair-scheduler.xml。在此文件里,设置各个队列的资源和规则。fair-scheduler.xml 的配置如下:<?xml version="1.0"?><allocations> <queue name="default"> <weight>1</weight> <schedulingPolicy>fair</schedulingPo原创 2021-11-05 15:13:45 · 530 阅读 · 0 评论 -
Hadoop 3.1 Capacity Scheduler Node Label 测试
<!-- root.a --> <property> <name>yarn.scheduler.capacity.root.a.queues</name> <value>aa, ab</value> <description> The queues at the this level (root is the root queue). </description>原创 2021-09-03 17:27:11 · 302 阅读 · 0 评论 -
SharedCacheManager does not start up caused by webapps/sharedcache not found in CLASSPATH
The webapp directory for the SharedCacheManager is missing and the SCM fails to start up with the following:2017-01-22 00:14:25,162 INFO org.apache.hadoop.service.AbstractService: Service SharedCa...原创 2018-03-21 17:50:41 · 288 阅读 · 0 评论 -
Enable HTTPS for YARN and MAPREDUCE2
Short Description:Steps to enable HTTPS for YARN and MAPREDUCE2Article1. As a first step, enable HTTPS for HDFS, you can follow the article https://community.hortonworks.com/articles/52875/enable-http...转载 2018-05-11 11:15:52 · 639 阅读 · 0 评论 -
Untangling Apache Hadoop YARN, Part 4: Fair Scheduler Queue Basics
In this installment, we provide insight into how the Fair Scheduler works, and why it works the way it does.In Part 3 of this series, you got a quick introduction to Fair Scheduler, one of the schedul...转载 2018-05-17 19:30:21 · 364 阅读 · 0 评论 -
Untangling Apache Hadoop YARN, Part 5: Using FairScheduler queue properties
Previously in Part 4, we described the most commonly used FairScheduler properties in Apache Hadoop. In Part 5, we’ll provide some examples to show how properties can be used, individually and in com...转载 2018-05-17 19:31:14 · 203 阅读 · 0 评论 -
Fairscheduler setting the user and group a queue can accept
If we set rule specified in queuePlacementPolicy, then the user can submit job to any queue.&amp;amp;amp;lt;queuePlacementPolicy&amp;amp;amp;gt; &amp;amp;amp;lt;rule name=&amp;amp;quot;specified&amp;amp;quot原创 2018-05-18 09:57:54 · 184 阅读 · 0 评论 -
Untangling Apache Hadoop YARN, Part 1: Cluster and YARN Basics
In this multipart series, fully explore the tangled ball of thread that is YARN.YARN (Yet Another Resource Negotiator) is the resource management layer for the Apache Hadoop ecosystem. YARN has been a...转载 2018-05-18 17:08:53 · 158 阅读 · 0 评论 -
The originalTrackingUrl changes in the process of application run
Before am register, the originalTrackingUrl=”N/A”; After am register, the originalTrackingUrl is the web url of application master, like ‘http://localhost:42298’private static final class AMRegis...原创 2018-07-11 14:43:26 · 376 阅读 · 0 评论 -
Why yarn reserve memory and CPU for application
An implementation detail of this change that prevents applications from starving under this new flexibility is the notion of reserved containers. Imagine two jobs are running that each have enough tas...转载 2018-12-26 15:24:09 · 606 阅读 · 0 评论 -
The process NodeManager clean Application log and intermediate data like map output
Content ListContent ListNodeStatusUpdaterImpl.startStatusUpdater.new Runnable.runCMgrCompletedAppsEvent’s event type is ContainerManagerEventType.FINISH_APPSNodeManager.serviceInitContainer...原创 2018-03-14 11:56:54 · 234 阅读 · 0 评论 -
YARN 状态机的原理和使用
有了异步处理,为什么还需要状态机?可以保存对象当前的状态。当接收到一个事件后,可以根据不同的状态,可以方便的决定怎么处理。 如接收到用户发送的KILL事件时: 当Job的状态为正在初始化时,只要把启动的进程kill就可以。 当Job的状态为正在运行,并且是提交到yarn的作业时,除了要把启动的进程kill,还需要调用yarn application -kill 把 yarn上运行的作...原创 2018-03-05 15:48:09 · 2443 阅读 · 0 评论 -
YARN Dispatcher的原理和应用
AsyncDispatcher ArchitectureAsyncDispatcher ExampleEnum AgentEventTypepublic enum AgentEventType { REGISTER, UNREGISTER, EXPIRED, HEARTBEAT, JOBLAUNCH, JOBKILL}...原创 2018-03-02 17:59:56 · 1168 阅读 · 0 评论 -
hadoop 2.6 Yarn Service源代码分析
Service有四种状态,用一个枚举来表示。/** * Service states */ public enum STATE { /** Constructed but not initialized */ NOTINITED(0, "NOTINITED"), /** Initialized but not started or stopped原创 2016-02-24 15:35:47 · 756 阅读 · 0 评论 -
hadoop 2.6 ApplicationMasterProtocol分析
ApplicationMasterProtocol是ApiplicationMaster和ResourceManager之间的接口,有3 个方法。/** * The protocol between a live instance of ApplicationMaster * and the ResourceManager. * * This is used by the原创 2016-05-11 16:52:54 · 752 阅读 · 0 评论 -
StateMachineFactory源代码详解
StateMachineFactory用于构建状态机拓扑。这个对象从语义上是不可改变的。如果你有一个StateMachineFactory对象,在API中没有改变他语义属性的操作。参数 状态机操作此对象类型。 实体的状态 自理的外部事件类型 事件对象/** * State machine topology. This object is semantically immu原创 2016-06-21 16:30:35 · 2177 阅读 · 0 评论 -
yarn 更換调度器之后,正在运行的作业可以自动拉起
1。原创 2016-07-26 18:34:59 · 641 阅读 · 0 评论 -
YARN Restapi : List running mapreduce apps which start before specified time
$resourcemanager-address}/ws/v1/cluster/apps?state=Running,Accepted&&applicationTypes=mapreduce&startedTimeEnd=1492593046942原创 2017-04-21 14:35:07 · 945 阅读 · 0 评论 -
SLS测试简介
SLS测试简介sls是Scheduler Load Simulator的缩写,是一个用来测试yarn调度器性能的工具。其实,若想完成测试,一共需要用到两个工具:Hadoop Rumen和SLS,下面逐个介绍。Hadoop RumenHadoop Rumen,是一个针对MR设计的日志解析分析工具。在SLS里,可以用这个工具解析mr jobhistory server的日志(可以解析一天的转载 2017-12-12 16:52:26 · 1334 阅读 · 3 评论 -
Upgrade YARN from Hadooop 2.6 to Hadoop 2.7
When resourcemanage is restarted, it may print the following error messages.2018-01-11 16:02:23,991 WARN org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=hadoop OPERATION=Applic原创 2018-01-12 14:47:27 · 450 阅读 · 0 评论 -
Container killed by the ApplicationMaster, Exit code is 143
之前发现在map任务里面经常看到Container killed by the ApplicationMaster,挺奇怪,不过任务最终是成功的,就没怎么管。不过最近测试集群跑的任务报143错误,还是重新看一下这个问题。分析版本:hadoop cdh5.4错误日志2015-12-30 17:31:09,994 INFO [AsyncDispatcher event handler]转载 2018-01-26 09:20:07 · 15256 阅读 · 2 评论 -
NodeManager Health Check 经验总结
一、 默认情况默认情况下,每个NdeManager自己的资源是在yarn-site.xml中配置的。如下面的配置时32个cpu cores,100G内存。现在集群中服务器cpu的线程数有32,40,48等几类。 property> name>yarn.nodemanager.resource.cpu-vcoresname> value>32value>原创 2018-01-28 15:20:39 · 1583 阅读 · 0 评论 -
NodeManager healthy check includes script health checker and local dir checker.
NodeMananger healthy check includes script health checker and local dir checker.By default, script health checker runs every 5 minutes, and local dir checker runss every 2 minutes.NodeHealthCheck...原创 2018-02-26 10:16:47 · 205 阅读 · 0 评论 -
hadoop 2.7.5 PublicLocalizer uses 4 threads by default to download resources.
LocalizerTrackerLocalizerTracker manages public localizer and private localizers. The public localizer is initialized when LocalizerTracker object is initialized, and The public localizer is star...原创 2018-02-28 15:01:52 · 256 阅读 · 0 评论 -
hadoop 2.7.5 LocalizerTracker uses seperate Private Localizer for every container
LocalizerTrackerLocalizerTracker manages both public public localizer and others. class LocalizerTracker extends AbstractService implements EventHandler<LocalizerEvent> { private final Pub...原创 2018-02-28 15:06:56 · 251 阅读 · 0 评论 -
hadoop 2.7.5 YARN ResouceManager Submit Application Process
1. Client calls submitApplication method of ClientRMService using RPC.2. ClientRMService. submitApplication calls RMAppManager.submitApplicationpublic SubmitApplicationResponse submitApplicat...原创 2018-01-29 17:00:02 · 1072 阅读 · 0 评论 -
YARN Service的原理及应用
Service TypeAbstratServiceCompositeServiceService Exampleclass Agent { private final int id; public Agent(int id) { this.id = id; } // hashCode and equals method mus...原创 2018-03-02 17:10:21 · 2003 阅读 · 1 评论 -
hadoop 2.6 AbstractYarnScheduler源代码解析
AbstractYarnScheduler 扩展了AbstractService 实现了 ResourceScheduler 。 SchedulerNode是AbstractYarnScheduler的一个参数化构建类,代表一个计算结点的各种属性,如下: availableResource:可用的资源。 usedResource:已经使用的资源; totalResour原创 2016-03-03 15:49:07 · 718 阅读 · 0 评论