Hive on Spark源码分析（五）—— RemoteDriver

Hive on Spark源码分析（一）—— SparkTask
Hive on Spark源码分析（二）—— SparkSession与HiveSparkClient
Hive on Spark源码分析（三）—— SparkClilent与SparkClientImpl（上）
Hive on Spark源码分析（四）—— SparkClilent与SparkClientImpl（下）
Hive on Spark源码分析（五）—— RemoteDriver
Hive on Spark源码分析（六）—— RemoteSparkJobMonitor与JobHandle 

RemoteDriver与SparkClient进行任务交互，并向Spark集群提交任务的。SparkClientImpl中通过调用RemoteDriver.main在新进程中启动了RemoteDriver

main函数

 
 
public static void main(String[] args) throws Exception {
    new RemoteDriver(args).run();
  }

run方法里主要做关闭线程，删除临时目录的工作

 
 
private void run() throws InterruptedException {
    synchronized (shutdownLock) {
      while (running) {
        shutdownLock.wait();
      }
    }
    executor.shutdownNow();
    try {
      FileUtils.deleteDirectory(localTmpDir);
    } catch (IOException e) {
      LOG.warn("Failed to delete local tmp dir: " + localTmpDir, e);
    }
  }

接下来我们看一下RemoteDriver的私有构造函数。处理参数，初始化环境变量，提交任务等都在私有工造方法中完成。首先是解析SparkClient传给RemoteDriver的参数，并付给相应的SparkConf：

  
  
    SparkConf conf = new SparkConf();
    String serverAddress = null;
    int serverPort = -1;
    for (int idx = 0; idx < args.length; idx += 2) {
      String key = args[idx];
      if (key.equals("--remote-host")) {
        serverAddress = getArg(args, idx);
      } else if (key.equals("--remote-port")) {
        serverPort = Integer.parseInt(getArg(args, idx));
      } else if (key.equals("--client-id")) {
        conf.set(SparkClientFactory.CONF_CLIENT_ID, getArg(args, idx));
      } else if (key.equals("--secret")) {
        conf.set(SparkClientFactory.CONF_KEY_SECRET, getArg(args, idx));
      } else if (key.equals("--conf")) {
        String[] val = getArg(args, idx).split("[=]", 2);
        conf.set(val[0], val[1]);
      } else {
        throw new IllegalArgumentException("Invalid command line: "
          + Joiner.on(" ").join(args));
      }
    }

后面的内容我们使用代码注释来分析一下：

   
   
    //线程池,用于创建线程执行任务
    executor = Executors.newCachedThreadPool();
    LOG.info("Connecting to: {}:{}", serverAddress, serverPort);
    //将RemoteDriver使用的参数保存到mapConf中
    Map<String, String> mapConf = Maps.newHashMap();
    for (Tuple2<String, String> e : conf.getAll()) {
      mapConf.put(e._1(), e._2());
      LOG.debug("Remote Driver configured with: " + e._1() + "=" + e._2());
    }
    // 得到clientId和secret用于与rpcServer建立连接时的认证.认证基于sasl,这里不考虑细节.
    // sasl作为pipeline中的一个handler实现
    String clientId = mapConf.get(SparkClientFactory.CONF_CLIENT_ID);
    Preconditions.checkArgument(clientId != null, "No client ID provided.");
    String secret = mapConf.get(SparkClientFactory.CONF_KEY_SECRET);
    Preconditions.checkArgument(secret != null, "No secret provided.");
    //获取hive.spark.client.rpc.threads的值,如果没有设置则获取到默认值8
    int threadCount = new RpcConfiguration(mapConf).getRpcThreadCount();
    this.egroup = new NioEventLoopGroup(
        threadCount,
        new ThreadFactoryBuilder()
            .setNameFormat("Driver-RPC-Handler-%d")
            .setDaemon(true)
            .build());
    //protocol实际是一个handler,与ClientProtocol类似
    this.protocol = new DriverProtocol();
    // The RPC library takes care of timing out this.
    // 这里的createClient返回一个Promise<Rpc>,从Future的get方法返回返回Rpc类型的对象
    // 这里会创建Bootstrap,connect到rpcServer
    this.clientRpc = Rpc.createClient(mapConf, egroup, serverAddress, serverPort,
      clientId, secret, protocol).get();
    this.running = true;

这里createClient中连接到rpcServer时处理timing out的方式跟SparkClientImpl中处理超时的方法一样，只是设置超时的参数不同。

我们来看一下createClient的具体实现。首先是获得Rpc的相关配置，得到hive.spark.client.connect.timeout属性的值赋给connectTimeoutMs

    
    
  public static Promise<Rpc> createClient(
      Map<String, String> config,
      final NioEventLoopGroup eloop,
      String host,
      int port,
      final String clientId,
      final String secret,
      final RpcDispatcher dispatcher) throws Exception {
    final RpcConfiguration rpcConf = new RpcConfiguration(config);
    int connectTimeoutMs = (int) rpcConf.getConnectTimeoutMs();

然后连接serverBootstrap，在前面的文章中有分析过，在HiveClient的进程中会创建RpcServer，即创建ServerBootstrap，这里的Bootstrap与其相连

     
     
    final ChannelFuture cf = new Bootstrap()
        .group(eloop)
        .handler(new ChannelInboundHandlerAdapter() { })
        .channel(NioSocketChannel.class)
        .option(ChannelOption.SO_KEEPALIVE, true)
        .option(ChannelOption.CONNECT_TIMEOUT_MILLIS, connectTimeoutMs)
        .connect(host, port);

下面处理timing out的方式与之前分析过的registerClient的处理方式相同

      
      
   // Set up a timeout to undo everything.
    final Runnable timeoutTask = new Runnable() {
      @Override
      public void run() {
        promise.setFailure(new TimeoutException("Timed out waiting for RPC server connection."));
      }
    };
    final ScheduledFuture<?> timeoutFuture = eloop.schedule(timeoutTask,
        rpcConf.getServerConnectTimeoutMs(), TimeUnit.MILLISECONDS);

下面我们回到RemoteDriver。接下来要为clientRpc添加监听器，在rpc关闭时关闭RemoteDriver

    
    
    this.clientRpc.addListener(new Rpc.Listener() {
      @Override
      public void rpcClosed(Rpc rpc) {
        LOG.warn("Shutting down driver because RPC channel was closed.");
        shutdown(null);
      }
    });

创建SparkContext，实例化JobsContext保存job执行时的运行时信息

     
     
    try {
      JavaSparkContext sc = new JavaSparkContext(conf);
      sc.sc().addSparkListener(new ClientListener());
      synchronized (jcLock) {
        //initiallize job context which holds runtime information about the job execution context.
        jc = new JobContextImpl(sc, localTmpDir);
        jcLock.notifyAll();
      }
    } catch (Exception e) {
      LOG.error("Failed to start SparkContext: " + e, e);
      //初始化sc,jc过程中,如果抛出异常时关闭发送错误,关闭各种服务
      shutdown(e);
      synchronized (jcLock) {
        jcLock.notifyAll();
      }
      throw e;
    }

因为在SparkContext还没就绪的时候提交的任务会被保存在一个等待队列jobQueued中，因此最后一步是将等待队列中的任务全部提交：

    
    
    synchronized (jcLock) {
      //提交所有等待队列里的任务
      for (Iterator<JobWrapper<?>> it = jobQueue.iterator(); it.hasNext();) {
        it.next().submit();
      }
    }
  }

至此RemoteDriver的构造就完成了，我们分别看一下其中用到的shutdown和submit方法。

shutdown方法用于停止RemoteDriver进程，参数为Throwable，正常停止调用时会传入null，否则传入一个error。在执行关闭动作之前，首先判断当前RemoteDriver的状态是否是running的，如果不是running的则不需要做任何操作。

    
    
  private synchronized void shutdown(Throwable error) {
    if (running) {
      if (error == null) {
        LOG.info("Shutting down remote driver.");
      } else {
        LOG.error("Shutting down remote driver due to error: " + error, error);
      }
      running = false;
      for (JobWrapper<?> job : activeJobs.values()) {
        cancelJob(job);
      }
      if (error != null) {
        protocol.sendError(error);
      }
      if (jc != null) {
        jc.stop();
      }
      clientRpc.close();
      egroup.shutdownGracefully();
      synchronized (shutdownLock) {
        shutdownLock.notifyAll();
      }
    }
  }

代码很简单，主要操作就是取消残留在activeJobs列表中的残留任务，停止jobContext，关闭rpc，关闭NioEventLoop。

submit方法负责job的提交执行。如果提交的时候job context还没有就绪，则现将job加入到等待队列中，否则直接调用JobWrapper的submit方法提交执行

   
   
  private void submit(JobWrapper<?> job) {
    synchronized (jcLock) {
      if (jc != null) {
        job.submit();
      } else {
        LOG.info("SparkContext not yet up, queueing job request.");
        jobQueue.add(job);
      }
    }
  }

在RemoteDriver中除了一些简单的私有属性外，还定义了三个重要的内部类：JobWrapper、ClientListener和DriverProtocol。在上面分析的RemoteDriver的构造过程中我们会发现，这三个内部类的使用贯穿在整个RemoteDriver的实现当中，因此我们首分别看一些这几个内部类的作用和实现。

1. JobWrapper

JobWrapper实现了Callable<Void>接口，再结合名字我们就很容易猜到该类是对job的封装，可以提交执行。 我们直接进入到JobWrapper中的方法，成员变量在方法中进行介绍。

首先作为一个callable的任务，核心方法就是call方法，我们来看一下JobWrapper中call方法的实现。

第一步是t通过protocol发送jobStarted消息

   
   
    @Override
    public Void call() throws Exception {
      protocol.jobStarted(req.id);

然后则是执行JobWrapper中封装的job自身的call方法

    
    
        T result = req.job.call(jc);

通过future判断job是否完成

     
     
        for (JavaFutureAction<?> future : jobs) {
          future.get();
          completed++;
          LOG.debug("Client job {}: {} of {} Spark jobs finished.",
              req.id, completed, jobs.size());
        }

接下来，如果job不为空，通过jobEndReceived的值来等待TaskEnd/JobEnd事件都已经处理完，因为 jobEndReceived只会在JobEnd事件的处理方法中通过调用jobDone来更新

     
     
        if (sparkJobId != null) {
          SparkJobInfo sparkJobInfo = jc.sc().statusTracker().getJobInfo(sparkJobId);
          if (sparkJobInfo != null && sparkJobInfo.stageIds() != null &&
              sparkJobInfo.stageIds().length > 0) {
            synchronized (jobEndReceived) {
              while (jobEndReceived.get() < jobs.size()) {
                jobEndReceived.wait();
              }
            }
          }
        }

然后获取sparkCoutner，发送jobFinished消息，结果置为前面job.call得到的result，错误置为null

      
      
        SparkCounters counters = null;
        if (sparkCounters != null) {
          counters = sparkCounters.snapshot();
        }
        protocol.jobFinished(req.id, result, null, counters);

如果过程中出现错误，则同样要发送JobFinished消息，只不过这次结果置为null，错误置为捕获到的t

      
      
} catch (Throwable t) {
        // Catch throwables in a best-effort to report job status back to the client. It\'s
        // re-thrown so that e executor can destroy the affected thread (or the JVM can
        // die or whatever would happen if the throwable bubbled up).
        LOG.info("Failed to run job " + req.id, t);
        protocol.jobFinished(req.id, null, t,
            sparkCounters != null ? sparkCounters.snapshot() : null);
        throw new ExecutionException(t);

最后在活跃job列表activeJobs中移除job，释放全部缓存的RDD

       
       
} finally {
        jc.setMonitorCb(null);
        activeJobs.remove(req.id);
        releaseCache();
      }
      return null;

以上就是call方法的内容。下面我们来看一下JobWrapper中的其他几个方法。

submit方法是供外部调用的提交任务的方法，它内部启动一个线程来提交执行JobWrapper

   
   
    void submit() {
      this.future = executor.submit(this);
    }

jobDone方法用于更新jobEndReceived

   
   
    void jobDone() {
      synchronized (jobEndReceived) {
        jobEndReceived.incrementAndGet();
        jobEndReceived.notifyAll();
      }
    }

当job完成后通过releaseCache方法释放缓存的RDD

   
   
    void releaseCache() {
      if (cachedRDDIds != null) {
        for (Integer cachedRDDId: cachedRDDIds) {
          jc.sc().sc().unpersistRDD(cachedRDDId, false);
        }
      }
    }

2. ClientListener

ClientListener继承自JavaSparkListener，用来监听来自Spark Scheduler的事件。ClientListener覆盖了父类中三个处理事件的方法。

当job开始时，触发onJobStart方法，将job的stage id和jobId保存到stageId这个hashmap中

   
   
    @Override
    public void onJobStart(SparkListenerJobStart jobStart) {
      synchronized (stageToJobId) {
        for (int i = 0; i < jobStart.stageIds().length(); i++) {
          stageToJobId.put((Integer) jobStart.stageIds().apply(i), jobStart.jobId());
        }
      }
    }

当job结束时，触发onJobEnd方法，首先从stageToJobId中删除该与该job对应的所有<stageId, jobId>

    
    
    @Override
    public void onJobEnd(SparkListenerJobEnd jobEnd) {
      synchronized (stageToJobId) {
        for (Iterator<Map.Entry<Integer, Integer>> it = stageToJobId.entrySet().iterator();
            it.hasNext();) {
          Map.Entry<Integer, Integer> e = it.next();
          if (e.getValue() == jobEnd.jobId()) {
            it.remove();
          }
        }
      }

然后获取该job对应的client的id，然后根据clientId，调用JobWrapper的jobDone方法到activeJobs中更新相应job的JobEndReceived

     
     
      String clientId = getClientId(jobEnd.jobId());
      if (clientId != null) {
        activeJobs.get(clientId).jobDone();
      }

当一个task结束时，触发onTaskEnd方法，通过protocol发送相应的task m etrics

     
     
    @Override
    public void onTaskEnd(SparkListenerTaskEnd taskEnd) {
      if (taskEnd.reason() instanceof org.apache.spark.Success$
          && !taskEnd.taskInfo().speculative()) {
        Metrics metrics = new Metrics(taskEnd.taskMetrics());
        Integer jobId;
        synchronized (stageToJobId) {
          jobId = stageToJobId.get(taskEnd.stageId());
        }
        // TODO: implement implicit AsyncRDDActions conversion instead of jc.monitor()?
        // TODO: how to handle stage failures?
        String clientId = getClientId(jobId);
        if (clientId != null) {
          protocol.sendMetrics(clientId, jobId, taskEnd.stageId(),
            taskEnd.taskInfo().taskId(), metrics);
        }
      }
    }

3. DriverProtocol

RemoteDriver中定义了一个主要的Handler：DriverProtocol extends BaseProtocol。DriverProtocol的实现整体与ClientProtocol类似，而且两者正是SparkClient与RemoteDriver实现互相通信的组件。DriverProtocol中定义了一些发送消息的方法，其实现同样是通过调用Rpc的call方法发送不同类型的消息，这些消息的类型恰好与ClientProtocol中的几个handle方法能够处理的消息类型吻合

   
   
  private class DriverProtocol extends BaseProtocol {
    //发送Error类型消息
    void sendError(Throwable error) {
      LOG.debug("Send error to Client: {}", Throwables.getStackTraceAsString(error));
      clientRpc.call(new Error(error));
    }
    //发送JobResult类型消息
    <T extends Serializable> void jobFinished(String jobId, T result,
        Throwable error, SparkCounters counters) {
      LOG.debug("Send job({}) result to Client.", jobId);
      clientRpc.call(new JobResult(jobId, result, error, counters));
    }
    //发送JobStarted类型消息
    void jobStarted(String jobId) {
      clientRpc.call(new JobStarted(jobId));
    }
    //发送JobSubmitted类型消息
    void jobSubmitted(String jobId, int sparkJobId) {
      LOG.debug("Send job({}/{}) submitted to Client.", jobId, sparkJobId);
      clientRpc.call(new JobSubmitted(jobId, sparkJobId));
    }
    //发送JobMetrics类型消息
    void sendMetrics(String jobId, int sparkJobId, int stageId, long taskId, Metrics metrics) {
      LOG.debug("Send task({}/{}/{}/{}) metric to Client.", jobId, sparkJobId, stageId, taskId);
      clientRpc.call(new JobMetrics(jobId, sparkJobId, stageId, taskId, metrics));
    }

而DriverProtocol中的handle方法所处理的消息类型刚好是SparkClient中发送消息的几个方法所发送的消息类型。

处理CancelJob类型消息时

   
   
    private void handle(ChannelHandlerContext ctx, CancelJob msg) {
      JobWrapper<?> job = activeJobs.get(msg.id);
      if (job == null || !cancelJob(job)) {
        LOG.info("Requested to cancel an already finished job.");
      }
    }

处理EndSession类型消息时，直接调用RemoteDriver的shutdown方法关闭RemoteDriver

   
   
    private void handle(ChannelHandlerContext ctx, EndSession msg) {
      LOG.debug("Shutting down due to EndSession request.");
      shutdown(null);
    }

处理JobRequest类型消息时，将消息封装到JobWrapper里，然后将JobWrapper添加到活跃任务列表activeJobs里并执行。关于提交任务时job context还没有就绪的情况的处理放在了submit方法中，因此这里不许要额外处理

    
    
    private void handle(ChannelHandlerContext ctx, JobRequest msg) {
      LOG.info("Received job request {}", msg.id);
      JobWrapper<?> wrapper = new JobWrapper<Serializable>(msg);
      activeJobs.put(msg.id, wrapper);
      submit(wrapper);
    }

处理SyncJobRequest类型消息时，需要先判断当前job context是否已经就绪，如果没有的话需要等待

     
     
    private Object handle(ChannelHandlerContext ctx, SyncJobRequest msg) throws Exception {
      // In case the job context is not up yet, let\'s wait, since this is supposed to be a
      // "synchronous" RPC.
      LOG.debug("liban: DriverProtocol received SyncJobRequest msg. waiting for jc to be up");
      if (jc == null) {
        synchronized (jcLock) {
          while (jc == null) {
            jcLock.wait();
            if (!running) {
              throw new IllegalStateException("Remote context is shutting down.");
            }
          }
        }
      }

然后添加Monitorcallback，直接调用job的call方法来执行任务

      
      
      jc.setMonitorCb(new MonitorCallback() {
        @Override
        public void call(JavaFutureAction<?> future,
            SparkCounters sparkCounters, Set<Integer> cachedRDDIds) {
          throw new IllegalStateException(
            "JobContext.monitor() is not available for synchronous jobs.");
        }
      });
      try {
        LOG.debug("liban: type of job in SyncJobRequest msg: " + msg.job.getClass().getSimpleName());
        return msg.job.call(jc);
      } finally {
        jc.setMonitorCb(null);
      }

这里我们会发现一点，同样是封装了job的消息，但是JobRequest和SyncJobRequest的处理方式却不同。原因就在于这两种消息是针对不同的job进行封装的。通过查找调用链发现， SyncJobRequest用来封装GetAppIDJob、GetStageInfoJob、AddjarJob、AddFileJob、GetExecutorCountJob、GetDefaultParallelismJob和GetJobInfo类型的job，这些类型的job可以快速执行结束，因此发送 SyncJobRequest的方法是在SparkClient中的run方法被调用，会在处理消息的方法的线程中直接执行。

而JobRequest中只封装一种类型的job：JobStatusJob。由于任务最后是通过调用job的call方法执行的，因此我们直接看一下这个job的call方法

   
   
    @Override
    public Serializable call(JobContext jc) throws Exception {
      JobConf localJobConf = KryoSerializer.deserializeJobConf(jobConfBytes);
      // Add jar to current thread class loader dynamically, and add jar paths to JobConf as Spark
      // may need to load classes from this jar in other threads.
      Map<String, Long> addedJars = jc.getAddedJars();
      if (addedJars != null && !addedJars.isEmpty()) {
        List<String> localAddedJars = SparkClientUtilities.addToClassPath(addedJars,
            localJobConf, jc.getLocalTmpDir());
        localJobConf.set(Utilities.HIVE_ADDED_JARS, StringUtils.join(localAddedJars, ";"));
      }
      // 反序列化出本地临时目录路径和SparkWork
      Path localScratchDir = KryoSerializer.deserialize(scratchDirBytes, Path.class);
      SparkWork localSparkWork = KryoSerializer.deserialize(sparkWorkBytes, SparkWork.class);
      logConfigurations(localJobConf);
      //获取sparkCounter
      SparkCounters sparkCounters = new SparkCounters(jc.sc());
      Map<String, List<String>> prefixes = localSparkWork.getRequiredCounterPrefix();
      if (prefixes != null) {
        for (String group : prefixes.keySet()) {
          for (String counterName : prefixes.get(group)) {
            sparkCounters.createCounter(group, counterName);
          }
        }
      }
      // 通过sparkCounter构造sparkReporter
      SparkReporter sparkReporter = new SparkReporter(sparkCounters);
      // Generate Spark plan
      SparkPlanGenerator gen =
        new SparkPlanGenerator(jc.sc(), null, localJobConf, localScratchDir, sparkReporter);
      SparkPlan plan = gen.generate(localSparkWork);
      // Execute generated plan.
      JavaPairRDD<HiveKey, BytesWritable> finalRDD = plan.generateGraph();
      // We use Spark RDD async action to submit job as it\'s the only way to get jobId now.
      JavaFutureAction<Void> future = finalRDD.foreachAsync(HiveVoidFunction.getInstance());
      jc.monitor(future, sparkCounters, plan.getCachedRDDIds());
      return null;
    }

我们可以看到，这个方法是最终会触发整个SparkWork的DAG执行的方法，也就是说JobStatusJob是包含了SparkWork主题的job类型，当前面的addjar，addfile等操作完成后，客户端会发送JobRequest类型消息，提交 JobStatusJob在Spark集群上执行。

至此，关于提交任务的几个重要组件就全部分析完了，从打开session到最后SparkWork变成RDD开始执行，整个执行链条也已经清晰了。