JobQueueTaskScheduler

最新推荐文章于 2021-03-18 22:29:42 发布
最新推荐文章于 2021-03-18 22:29:42 发布
阅读量620 收藏
点赞数
分类专栏： Hadoop
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/lizhe10177/article/details/18869123
版权

注：我们以map任务为例，下面默认的任务都为map任务，任务数量也是map任务数量。

其中的主要部分在于assignTasks函数。它的主要功能是：先得到TaskTracker的可用槽数，可用总槽数减去已用槽数，然后不断从JobTracker的待调度的作业中选择优先级别最高的作业来调度。当然这仅仅是我们的设想，而在JobQueueTaskScheduler的assignTasks函数实现中，有几个细节需要说明：

1 可用槽数

可用槽数并不是TaskTracker的最大槽数减去已用槽数，看下面

int availableMapSlots = trackerCurrentMapCapacity - trackerRunningMaps;

 

final int trackerCurrentMapCapacity = 
      Math.min((int)Math.ceil(mapLoadFactor * trackerMapCapacity), 
                              trackerMapCapacity);

 

 mapLoadFactor = (double)remainingMapLoad / clusterMapCapacity;


其中remainingMapLoad是集群的所有待调度任务数。clusterMapCapacity是集群总槽数。经过mapLoadFactor这个参数的调节，可以使每个发送请求的TaskTracker最终的负载大致跟相当，从而避免负载失衡。

2 预留槽数

集群上会预留一部分槽供一些任务重启。故在分配任务时增加了一个exceededPadding布尔变量，用以指示是否超出了预留界限。如果超过则停止分配任务。exceededPadding的计算：

        if (totalTasks + padding >= totalTaskCapacity) {
          exceededPadding = true;
          break;
        }

tatalTasks是集群的总任务数，totalTaskCapacity是集群总槽数

    int totalTasks = 
      (isMapTask) ? clusterStatus.getMapTasks() : 
        clusterStatus.getReduceTasks();
    int totalTaskCapacity = 
      isMapTask ? clusterStatus.getMaxMapTasks() : 
                  clusterStatus.getMaxReduceTasks();

maxTaskTrackerSlots是TaskTracker的总槽数。

totalNeededTasks 是集群里所有待调度的任务， padFraction是用户配置参数。

          padding = 
            Math.min(maxTaskTrackerSlots,
                     (int) (totalNeededTasks * padFraction));


在此不深究padding个数设置的原理，或许有更好的计算方式，以下为JobQueueTaskScheduler的全部代码：

 

/**
 * Licensed to the Apache Software Foundation (ASF) under one
 * or more contributor license agreements.  See the NOTICE file
 * distributed with this work for additional information
 * regarding copyright ownership.  The ASF licenses this file
 * to you under the Apache License, Version 2.0 (the
 * "License"); you may not use this file except in compliance
 * with the License.  You may obtain a copy of the License at
 *
 *     http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */
package org.apache.hadoop.mapred;

import java.io.IOException;
import java.util.ArrayList;
import java.util.Collection;
import java.util.List;

import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.apache.hadoop.conf.Configuration;

/**
 * A {@link TaskScheduler} that keeps jobs in a queue in priority order (FIFO
 * by default).
 */
class JobQueueTaskScheduler extends TaskScheduler {
  
  private static final int MIN_CLUSTER_SIZE_FOR_PADDING = 3;
  public static final Log LOG = LogFactory.getLog(JobQueueTaskScheduler.class);
  
  protected JobQueueJobInProgressListener jobQueueJobInProgressListener;
  protected EagerTaskInitializationListener eagerTaskInitializationListener;
  private float padFraction;
  
  public JobQueueTaskScheduler() {
    this.jobQueueJobInProgressListener = new JobQueueJobInProgressListener();
  }
  
  @Override
  public synchronized void start() throws IOException {
    super.start();
    taskTrackerManager.addJobInProgressListener(jobQueueJobInProgressListener);
    eagerTaskInitializationListener.setTaskTrackerManager(taskTrackerManager);
    eagerTaskInitializationListener.start();
    taskTrackerManager.addJobInProgressListener(
        eagerTaskInitializationListener);
  }
  
  @Override
  public synchronized void terminate() throws IOException {
    if (jobQueueJobInProgressListener != null) {
      taskTrackerManager.removeJobInProgressListener(
          jobQueueJobInProgressListener);
    }
    if (eagerTaskInitializationListener != null) {
      taskTrackerManager.removeJobInProgressListener(
          eagerTaskInitializationListener);
      eagerTaskInitializationListener.terminate();
    }
    super.terminate();
  }
  
  @Override
  public synchronized void setConf(Configuration conf) {
    super.setConf(conf);
    padFraction = conf.getFloat("mapred.jobtracker.taskalloc.capacitypad", 
                                 0.01f);
    this.eagerTaskInitializationListener =
      new EagerTaskInitializationListener(conf);
  }

  @Override
  public synchronized List<Task> assignTasks(TaskTrackerStatus taskTracker)
      throws IOException {

    ClusterStatus clusterStatus = taskTrackerManager.getClusterStatus();
    final int numTaskTrackers = clusterStatus.getTaskTrackers();
    final int clusterMapCapacity = clusterStatus.getMaxMapTasks();
    final int clusterReduceCapacity = clusterStatus.getMaxReduceTasks();

    Collection<JobInProgress> jobQueue =
      jobQueueJobInProgressListener.getJobQueue();

    //
    // Get map + reduce counts for the current tracker.
    //
    final int trackerMapCapacity = taskTracker.getMaxMapTasks();
    final int trackerReduceCapacity = taskTracker.getMaxReduceTasks();
    final int trackerRunningMaps = taskTracker.countMapTasks();
    final int trackerRunningReduces = taskTracker.countReduceTasks();

    // Assigned tasks
    List<Task> assignedTasks = new ArrayList<Task>();

    //
    // Compute (running + pending) map and reduce task numbers across pool
    //
    int remainingReduceLoad = 0;
    int remainingMapLoad = 0;
    synchronized (jobQueue) {
      for (JobInProgress job : jobQueue) {
        if (job.getStatus().getRunState() == JobStatus.RUNNING) {
          remainingMapLoad += (job.desiredMaps() - job.finishedMaps());
          if (job.scheduleReduces()) {
            remainingReduceLoad += 
              (job.desiredReduces() - job.finishedReduces());
          }
        }
      }
    }

    // Compute the 'load factor' for maps and reduces
    double mapLoadFactor = 0.0;
    if (clusterMapCapacity > 0) {
      mapLoadFactor = (double)remainingMapLoad / clusterMapCapacity;
    }
    double reduceLoadFactor = 0.0;
    if (clusterReduceCapacity > 0) {
      reduceLoadFactor = (double)remainingReduceLoad / clusterReduceCapacity;
    }
        
    //
    // In the below steps, we allocate first map tasks (if appropriate),
    // and then reduce tasks if appropriate.  We go through all jobs
    // in order of job arrival; jobs only get serviced if their 
    // predecessors are serviced, too.
    //

    //
    // We assign tasks to the current taskTracker if the given machine 
    // has a workload that's less than the maximum load of that kind of
    // task.
    // However, if the cluster is close to getting loaded i.e. we don't
    // have enough _padding_ for speculative executions etc., we only 
    // schedule the "highest priority" task i.e. the task from the job 
    // with the highest priority.
    //
    
    final int trackerCurrentMapCapacity = 
      Math.min((int)Math.ceil(mapLoadFactor * trackerMapCapacity), 
                              trackerMapCapacity);
    int availableMapSlots = trackerCurrentMapCapacity - trackerRunningMaps;
    boolean exceededMapPadding = false;
    if (availableMapSlots > 0) {
      exceededMapPadding = 
        exceededPadding(true, clusterStatus, trackerMapCapacity);
    }
    
    int numLocalMaps = 0;
    int numNonLocalMaps = 0;
    scheduleMaps:
    for (int i=0; i < availableMapSlots; ++i) {
      synchronized (jobQueue) {
        for (JobInProgress job : jobQueue) {
          if (job.getStatus().getRunState() != JobStatus.RUNNING) {
            continue;
          }

          Task t = null;
          
          // Try to schedule a node-local or rack-local Map task
          t = 
            job.obtainNewLocalMapTask(taskTracker, numTaskTrackers,
                                      taskTrackerManager.getNumberOfUniqueHosts());
          if (t != null) {
            assignedTasks.add(t);
            ++numLocalMaps;
            
            // Don't assign map tasks to the hilt!
            // Leave some free slots in the cluster for future task-failures,
            // speculative tasks etc. beyond the highest priority job
            if (exceededMapPadding) {
              break scheduleMaps;
            }
           
            // Try all jobs again for the next Map task 
            break;
          }
          
          // Try to schedule a node-local or rack-local Map task
          t = 
            job.obtainNewNonLocalMapTask(taskTracker, numTaskTrackers,
                                   taskTrackerManager.getNumberOfUniqueHosts());
          
          if (t != null) {
            assignedTasks.add(t);
            ++numNonLocalMaps;
            
            // We assign at most 1 off-switch or speculative task
            // This is to prevent TaskTrackers from stealing local-tasks
            // from other TaskTrackers.
            break scheduleMaps;
          }
        }
      }
    }
    int assignedMaps = assignedTasks.size();

    //
    // Same thing, but for reduce tasks
    // However we _never_ assign more than 1 reduce task per heartbeat
    //
    final int trackerCurrentReduceCapacity = 
      Math.min((int)Math.ceil(reduceLoadFactor * trackerReduceCapacity), 
               trackerReduceCapacity);
    final int availableReduceSlots = 
      Math.min((trackerCurrentReduceCapacity - trackerRunningReduces), 1);
    boolean exceededReducePadding = false;
    if (availableReduceSlots > 0) {
      exceededReducePadding = exceededPadding(false, clusterStatus, 
                                              trackerReduceCapacity);
      synchronized (jobQueue) {
        for (JobInProgress job : jobQueue) {
          if (job.getStatus().getRunState() != JobStatus.RUNNING ||
              job.numReduceTasks == 0) {
            continue;
          }

          Task t = 
            job.obtainNewReduceTask(taskTracker, numTaskTrackers, 
                                    taskTrackerManager.getNumberOfUniqueHosts()
                                    );
          if (t != null) {
            assignedTasks.add(t);
            break;
          }
          
          // Don't assign reduce tasks to the hilt!
          // Leave some free slots in the cluster for future task-failures,
          // speculative tasks etc. beyond the highest priority job
          if (exceededReducePadding) {
            break;
          }
        }
      }
    }
    
    if (LOG.isDebugEnabled()) {
      LOG.debug("Task assignments for " + taskTracker.getTrackerName() + " --> " +
                "[" + mapLoadFactor + ", " + trackerMapCapacity + ", " + 
                trackerCurrentMapCapacity + ", " + trackerRunningMaps + "] -> [" + 
                (trackerCurrentMapCapacity - trackerRunningMaps) + ", " +
                assignedMaps + " (" + numLocalMaps + ", " + numNonLocalMaps + 
                ")] [" + reduceLoadFactor + ", " + trackerReduceCapacity + ", " + 
                trackerCurrentReduceCapacity + "," + trackerRunningReduces + 
                "] -> [" + (trackerCurrentReduceCapacity - trackerRunningReduces) + 
                ", " + (assignedTasks.size()-assignedMaps) + "]");
    }

    return assignedTasks;
  }

  private boolean exceededPadding(boolean isMapTask, 
                                  ClusterStatus clusterStatus, 
                                  int maxTaskTrackerSlots) { 
    int numTaskTrackers = clusterStatus.getTaskTrackers();
    int totalTasks = 
      (isMapTask) ? clusterStatus.getMapTasks() : 
        clusterStatus.getReduceTasks();
    int totalTaskCapacity = 
      isMapTask ? clusterStatus.getMaxMapTasks() : 
                  clusterStatus.getMaxReduceTasks();

    Collection<JobInProgress> jobQueue =
      jobQueueJobInProgressListener.getJobQueue();

    boolean exceededPadding = false;
    synchronized (jobQueue) {
      int totalNeededTasks = 0;
      for (JobInProgress job : jobQueue) {
        if (job.getStatus().getRunState() != JobStatus.RUNNING ||
            job.numReduceTasks == 0) {
          continue;
        }

        //
        // Beyond the highest-priority task, reserve a little 
        // room for failures and speculative executions; don't 
        // schedule tasks to the hilt.
        //
        totalNeededTasks += 
          isMapTask ? job.desiredMaps() : job.desiredReduces();
        int padding = 0;
        if (numTaskTrackers > MIN_CLUSTER_SIZE_FOR_PADDING) {
          padding = 
            Math.min(maxTaskTrackerSlots,
                     (int) (totalNeededTasks * padFraction));
        }
        if (totalTasks + padding >= totalTaskCapacity) {
          exceededPadding = true;
          break;
        }
      }
    }

    return exceededPadding;
  }

  @Override
  public synchronized Collection<JobInProgress> getJobs(String queueName) {
    return jobQueueJobInProgressListener.getJobQueue();
  }  
}

知乎哲也
关注
专栏目录
Hadoop调度算法
08-10
当TaskTracker有空闲资源时，JobQueueTaskScheduler会从队列头部选择优先级最高的作业（即最早提交的作业）分配任务。该算法简单易实现，但可能导致长作业等待时间过长，短作业响应迅速，可能会影响系统的整体效率。...
JobTracker任务调度器之JobQueueTaskScheduler
xhh198781的专栏
12-06 8427
在客户端把作业提交给JobTracker节点之后，JobTracker节点就可以使用任务调度器TaskScheduler将这个Job的任务分配给那些合适的TaskTracker节点来执行。当然在JobTracker调度该Job之前，必须要确保该Job的JobInProgress被初始化了，即将Job划分为若干个map任务和reduce任务。在JobTracker中有一个基于优先级的FIFO任务调度
【Hadoop】JobQueueTaskScheduler
TragicJun的专栏
11-08 1118
JobQueueTaskScheduler是JT默认的任务调度器，其本质上维护一个priority-based FIFO作业队列，基本的机制是优先调度高优先级/提交时间早的作业。但是，具体的调度策略需要综合考虑一系列的问题，比如cluster资源利用率、系统吞吐率、任务locality等等。下面从这些方面来分析一下调度策略。 单个TaskTracker利用率 最初，JT调度器每次h
Hadoop的默认调度器JobQueueTaskScheduler的一个Map Task Pending问题
PunC的专栏
03-07 2166
在正式环境的Hadoop任务调度里，集群整体资源还有大量slot的情况下，出现了某些MapReduce任务无法被调度，一直处于pending状态，无法获得集群slot资源进行计算的诡异情况。更改几个指定类的loglevel后，查看其Debug的log发现了问题原因：这是由于Hadoop 1.x默认的FIFO调度器JobQueueTaskScheduler在进行集群均衡计算时，忽略Speculati
个人面试题总结
weixin_49196365的博客
03-18 7276
操作系统 用户态和内核态 程序和进程的区别 进程和线程 多进程和多线程的区别 线程有阻塞和唤醒吗 有，得不到cpu就会阻塞，得不到系统资源会导致进程阻塞 为什么线程切换花销小 进程切换时要切页表，而且往往伴随着页调度 统一进程下的线程共享进程里的资源，线程只需要保存线程的上下文（相关寄存器状态和栈的信息） 堆栈的区别， 堆是线程公有的 栈是线程私有的 在多线程环境下编程需要注意哪些情况呢 要避免产生死锁 避免死锁都有哪些方式方法 静态库和动态库的区别？ 静态链接和动态链接呢？ 虚拟地址有什么好处 虚拟内存页
hadoop学习源码学习（二）
05-13
#### 默认调度器JobQueueTaskScheduler的实现 1. **检查TaskTracker资源利用率**： - 调度器在分配任务前，首先检查TaskTracker端还能承担多少Map和Reduce任务。 - 检查任务数量是否超出TaskTracker的最大限制和...
hadoop源码之JobQueueTaskScheduler
luyee2010的专栏
04-15 1437
class JobQueueTaskScheduler extends TaskScheduler ---------FIFO 1，start()方法 public synchronized void start() throws IOException {     super.start();     taskTrackerManager.addJobInProgressListener
由TaskTracker启动到Task执行
lizhe10177的专栏
01-28 1072
1 TaskTracker main函数   public static void main(String argv[]) throws Exception { StringUtils.startupShutdownMessage(TaskTracker.class, argv, LOG); if (argv.length != 0) { System.out.pr
JobInProgress初始化为TaskInProgress
lizhe10177的专栏
02-06 1061
在http://blog.csdn.net/lizhe10177/article/details/17763879这篇文章中曾跟踪作业的提交，最终是通过listener.jobAdded(job)将作业提交到listener的初始化队列里 public void jobAdded(JobInProgress job) { synchronized (jobInitQueue) {
JobTracker
lizhe10177的专栏
02-02 800
从main方法开始跟踪 1 main main的关键代码有两句，创建了实例并调用offerService方法。 JobTracker tracker = startTracker(new JobConf()); tracker.offerService(); 2  startTracker main方法中的startTracker调用了它的一个重载函数，该函数的主要代码
Hadoop作业提交跟踪
lizhe10177的专栏
01-02 791
1 JobClient.runJob 在配置好作业后，我们通过JobClient.runJob(conf)提交作业，从这开始 JobClient jc = new JobClient(job); RunningJob rj = jc.submitJob(job); 该方法首先实例化一个JobClient，但后调用submitJob（job） 2 submitJob
Hadoop源码之Configuration类
lizhe10177的专栏
01-03 654
1 Configuration概况 Configuration是Hadoop中与配置文件打交道的类，他也是JobConfiguration的父类。配置文件一般是内容为一系列xml数据格式的键值对。配置文件可以由一个字符串指定，也可以由一个Path实例指定。 Hadoop默认情况下会加载两个配置文件，core-default.xml和core-site.xml，用户可以为程序加载额外的配置文件。
TaskTracker
lizhe10177的专栏
01-29 509
main函数 JobTracker实现了线程接口，main方法创建一个JobTracker实例并且启动该线程 JobConf conf=new JobConf(); // enable the server to track time spent waiting on locks ReflectionUtils.setContentionTracing
JobQueueJobInProgressListener
lizhe10177的专栏
01-28 449
JobQueueJobInProgressListener继承自抽象类JobInprogressListener /** * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file * distribut
Python 实现 LSTM 和 XGBoost 组合模型来预测 Apple Inc.（AAPL）股票价格（包含详细的完整的程序
10-06
内容概要：详细演示了使用 Python 中的 LSTM 和 XGBoost 结合来创建股票价格预测模型的方法。该示例介绍了从数据提取到模型优化全过程的操作，并最终通过图形比较预测值和真实值，展示模型的有效性，有助于提高金融投资决策水平和风险管理能力。本项目的亮点之一就是它融合 LSTM 捕获时间关系的强大能力和 XGBoost 在复杂特征之间的建模优势。 适用人群：有Python编程经验的人士以及金融市场投资者和技术分析师。 使用场景及目标：应用于金融市场的投资策略规划，特别是针对需要长期监控、短期交易决策的股票，用于辅助进行市场走势判断和交易决策支持。 额外信息：此外还包括对未来工作的改进建议：加入更多金融技术指标的考量以及使用更高级机器学习模型的可能性。
2019年计算机网络专业学生实习总结(二篇).pdf
10-06
计算机试题试卷课件
222222222222222222222222222222222222222222222
最新发布
10-06
222222222222222222222222222222222222222222222
2020年计算机机试和笔试01至考试题.pdf
10-06
计算机试题试卷课件
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值