TaskTracker

最新推荐文章于 2021-12-02 17:44:37 发布

知乎哲也

最新推荐文章于 2021-12-02 17:44:37 发布

阅读量516

点赞数

分类专栏： Hadoop

本文链接：https://blog.csdn.net/lizhe10177/article/details/18866889

版权

Hadoop 专栏收录该内容

14 篇文章 0 订阅

订阅专栏

main函数

JobTracker实现了线程接口，main方法创建一个JobTracker实例并且启动该线程

JobConf conf=new JobConf();
      // enable the server to track time spent waiting on locks
      ReflectionUtils.setContentionTracing
        (conf.getBoolean("tasktracker.contention.tracking", false));
      new TaskTracker(conf).run();

构造方法

TaskTracker在构造方法中先初始化部分变量，并在构造方法里调用initialize函数，

 this.server = new HttpServer("task", httpBindAddress, httpPort,
        httpPort == 0, conf);

initialize();

initialize函数

initialize函数完成reportServer的创建，负责向JobTracker报告状态；jobClient的创建，负责向JobTracker索要任务；然后创建一个mapLauncher和reduceLauncher并启动，负责启动map任务或者reduce任务。

 this.taskReportServer =
      RPC.getServer(this, bindAddress, tmpPort, 2 * max, false, this.fConf);
    this.taskReportServer.start();

this.jobClient = (InterTrackerProtocol) 
      RPC.waitForProxy(InterTrackerProtocol.class,
                       InterTrackerProtocol.versionID, 
                       jobTrackAddr, this.fConf);

    mapLauncher = new TaskLauncher(maxCurrentMapTasks);
    reduceLauncher = new TaskLauncher(maxCurrentReduceTasks);
    mapLauncher.start();
    reduceLauncher.start();

run函数

首先启动了清理线程，该线程不断从待清理的TaskTrackerAction队列里取出Action并处理。

startCleanupThreads();

调用了offerservice函数

offerService函数

offerService函数通过transmitHeartBeat传送心跳并获得任务

HeartbeatResponse heartbeatResponse = transmitHeartBeat(now);

获取任务后，根据任务的属性放到相应的队列，并等待后台的mapLauncher，reduceLauncher，taskCleanupThread等线程处理。注意到tasksToCleanup队列就是介绍run函数一节中提到的清理队列。

TaskTrackerAction[] actions = heartbeatResponse.getActions();

if (actions != null){ 
          for(TaskTrackerAction action: actions) {
            if (action instanceof LaunchTaskAction) {
              addToTaskQueue((LaunchTaskAction)action);
            } else if (action instanceof CommitTaskAction) {
              CommitTaskAction commitAction = (CommitTaskAction)action;
              if (!commitResponses.contains(commitAction.getTaskID())) {
                LOG.info("Received commit task action for " + 
                          commitAction.getTaskID());
                commitResponses.add(commitAction.getTaskID());
              }
            } else {
              tasksToCleanup.put(action);
            }
          }
        }

transmitHeartBeat函数

transmitHeartBeat正是通过jobClient的远程调用返回HeartbeatResponse

 HeartbeatResponse heartbeatResponse = jobClient.heartbeat(status, 
                                                              justStarted,
                                                              justInited,
                                                              askForNewTask, 
                                                              heartbeatResponseId);