ccah-500 第5题 How will the Fair Scheduler handle these two jobs?

原创 2016年06月01日 10:11:44

5.You have a cluster running with the fair Scheduler enabled. There are currently no jobs running on the cluster, and you submit a job A, so that only job A is running on the cluster. A while later, you submit Job B. now Job A and Job B are running on the cluster at the same time. How will the Fair Scheduler handle these two jobs? 

A. When Job B gets submitted, it will get assigned tasks, while job A continues to run with fewer tasks. 

B. When Job B gets submitted, Job A has to finish first, before job B can gets scheduled. 

C. When Job A gets submitted, it doesn't consumes all the task slots. 

D. When Job A gets submitted, it consumes all the task slots. 

Answer: B --> A

解析: A



With the Fair Scheduler (iii in Figure 4-3), there is no need to reserve a set amount of

capacity, since it will dynamically balance resources between all running jobs. Just after

the first (large) job starts, it is the only job running, so it gets all the resources in the

cluster. When the second (small) job starts, it is allocated half of the cluster resources so

that each job is using its fair share of resources.

Note that there is a lag between the time the second job starts and when it receives its fair

share, since it has to wait for resources to free up as containers used by the first job

complete. After the small job completes and no longer requires resources, the large job

goes back to using the full cluster capacity again. The overall effect is both high cluster

utilization and timely small job completion.


Fair scheduling is a method of assigning resources to jobs such that all jobs get, on average, an equal share of resources over time. When there is a single job running, that job uses the entire cluster. When other jobs are submitted, tasks slots that free up are assigned to the new jobs, so that each job gets roughly the same amount of CPU time. Unlike the default Hadoop scheduler, which forms a queue of jobs, this lets short jobs finish in reasonable time while not starving long jobs. It is also a reasonable way to share a cluster between a number of users.




spark 内置TaskScheduler级别的调度算法,分别是队列与公平,默认是队列方式。具体源码可参考SchedulableBuilder 准确来讲spark有两层公平,一层是公平调度,一层是公平...
  • w412692660
  • w412692660
  • 2015年03月16日 18:31
  • 2617

Fair Scheduler

与Capacity Scheduler类似,Fair Scheduler以队列为单位划分资源,每个队列可设定一定比例的资源最低保证和使用上限,同时,每个用户也可设定一定的资源使用上限以防止资源滥用;当...
  • a118170653
  • a118170653
  • 2015年02月10日 11:00
  • 978

yarn Fairscheduler与Capacityscheduler

  • Nougats
  • Nougats
  • 2017年05月01日 20:29
  • 1156

Linux 内核 Completely Fair Scheduler (cfs)调度算法

linux内核调度在2.6.23 之前使用的大名鼎鼎的O(1)算法。O(1) 调度器跟踪运行队列中可运行的任务(实际上,每个优先级水平有两个运行队列 — 一个用于活动任务,一个用于过期任务), 这意味...
  • haidao2009
  • haidao2009
  • 2013年05月24日 15:59
  • 2021

yarn fair scheduler 之概述

所有的调度器都是基于事件模型,需要处理的一个重要事件是nodeUpdate,处理各个节点的心跳。 对于公平调度器,在处理某一Node心跳事件时,先处理刚刚启动的Container和已完成任务的Con...
  • wujun8
  • wujun8
  • 2014年06月19日 16:52
  • 1848

Spark Job Scheduling

Overview Spark为计算任务提供资源调度,Spark App运行在独立的一组Executor线程之上,Spark调度器可以提供应用之间的资源调度。其次,在Spark App中,会有多个...
  • Dax1n
  • Dax1n
  • 2017年04月15日 13:56
  • 474

数据库---Oracle Scheduler Jobs 基本使用

  • tianya9006
  • tianya9006
  • 2015年11月12日 15:27
  • 782


158.You create two resource plans, one for data warehouse loading jobs at night and the other for a...
  • rlhua
  • rlhua
  • 2013年10月26日 16:26
  • 6042

配置hadoop 使用fair scheduler调度器

hadoop版本为cloudera hadoop cdh3u3 配置步骤为 1.  将$HADOOP_HOME/contrib/fairscheduler/hadoop-fairsched...
  • xiaolang85
  • xiaolang85
  • 2014年03月03日 21:53
  • 2837

Hadoop权威指南读书笔记(2) — Yarn简介及Capacity & Fair Scheduler

Yarn简介Yarn(Yet Another Resource Negotiator)是hadoop2中引入的一个集群资源管理系统。...
  • labud
  • labud
  • 2016年07月27日 19:18
  • 1279
您举报文章:ccah-500 第5题 How will the Fair Scheduler handle these two jobs?