spark task启动

原创 2015年11月19日 14:39:31
    woker接受到LaunchTask消息后会启动task,以local模式为例,代码如下。task是由executor来启动的
override def receiveWithLogging = {
  case ReviveOffers =>
    reviveOffers()
    ......

def reviveOffers() {
  val offers = Seq(new WorkerOffer(localExecutorId, localExecutorHostname, freeCores))
  val tasks = scheduler.resourceOffers(offers).flatten
  //把task循环启动起来
  for (task <- tasks) {
    freeCores -= scheduler.CPUS_PER_TASK
    executor.launchTask(executorBackend, taskId = task.taskId, attemptNumber = task.attemptNumber,
      task.name, task.serializedTask)
  }
  if (tasks.isEmpty && scheduler.activeTaskSets.nonEmpty) {
    // Try to reviveOffer after 1 second, because scheduler may wait for locality timeout
    context.system.scheduler.scheduleOnce(1000 millis, self, ReviveOffers)
  }
}

launchTask会将task封装为TaskRunner,放入线程池中运行,此时task是被序列化的
def launchTask(
    context: ExecutorBackend,
    taskId: Long,
    attemptNumber: Int,
    taskName: String,
    serializedTask: ByteBuffer) {
  val tr = new TaskRunner(context, taskId = taskId, attemptNumber = attemptNumber, taskName,
    serializedTask)
  runningTasks.put(taskId, tr)
  threadPool.execute(tr)
}
TaskRunner是一个实现Runnable的线程执行体,被放入线程池后续先进性反序列化,然后执行该task,汇报task状态为running
override def run() {
    val deserializeStartTime = System.currentTimeMillis()
    Thread.currentThread.setContextClassLoader(replClassLoader)
    val ser = env.closureSerializer.newInstance()
    logInfo(s"Running $taskName (TID $taskId)")
    execBackend.statusUpdate(taskId, TaskState.RUNNING, EMPTY_BYTE_BUFFER)
    var taskStart: Long = 0
    startGCTime = gcTime


    try {
    	//反序列化
      val (taskFiles, taskJars, taskBytes) = Task.deserializeWithDependencies(serializedTask)
      updateDependencies(taskFiles, taskJars)
      task = ser.deserialize[Task[Any]](taskBytes, Thread.currentThread.getContextClassLoader)


      // If this task has been killed before we deserialized it, let's quit now. Otherwise,
      // continue executing the task.
      if (killed) {
        // Throw an exception rather than returning, because returning within a try{} block
        // causes a NonLocalReturnControl exception to be thrown. The NonLocalReturnControl
        // exception will be caught by the catch block, leading to an incorrect ExceptionFailure
        // for the task.
        throw new TaskKilledException
      }


      attemptedTask = Some(task)
      logDebug("Task " + taskId + "'s epoch is " + task.epoch)
      env.mapOutputTracker.updateEpoch(task.epoch)


      // 开始执行task
      taskStart = System.currentTimeMillis()
      val value = task.run(taskAttemptId = taskId, attemptNumber = attemptNumber)
      val taskFinish = System.currentTimeMillis()


      // If the task has been killed, let's fail it.
      if (task.killed) {
        throw new TaskKilledException
      }
      ......
      val directResult = new DirectTaskResult(valueBytes, accumUpdates, task.metrics.orNull)
      val serializedDirectResult = ser.serialize(directResult)
      val resultSize = serializedDirectResult.limit


      // directSend = sending directly back to the driver
      val serializedResult = {
        if (maxResultSize > 0 && resultSize > maxResultSize) {
          logWarning(s"Finished $taskName (TID $taskId). Result is larger than maxResultSize " +
            s"(${Utils.bytesToString(resultSize)} > ${Utils.bytesToString(maxResultSize)}), " +
            s"dropping it.")
          ser.serialize(new IndirectTaskResult[Any](TaskResultBlockId(taskId), resultSize))
        } else if (resultSize >= akkaFrameSize - AkkaUtils.reservedSizeBytes) {
          val blockId = TaskResultBlockId(taskId)
          env.blockManager.putBytes(
            blockId, serializedDirectResult, StorageLevel.MEMORY_AND_DISK_SER)
          logInfo(
            s"Finished $taskName (TID $taskId). $resultSize bytes result sent via BlockManager)")
          ser.serialize(new IndirectTaskResult[Any](blockId, resultSize))
        } else {
          logInfo(s"Finished $taskName (TID $taskId). $resultSize bytes result sent to driver")
          serializedDirectResult
        }
      }
      execBackend.statusUpdate(taskId, TaskState.FINISHED, serializedResult)
    } catch {
      .....
    } finally {
      // Release memory used by this thread for shuffles
      env.shuffleMemoryManager.releaseMemoryForThisThread()
      // Release memory used by this thread for unrolling blocks
      env.blockManager.memoryStore.releaseUnrollMemoryForThisThread()
      // Release memory used by this thread for accumulators
      Accumulators.clear()
      runningTasks.remove(taskId)
    }
  }
}  
  


相关文章推荐

spark源码学习(八)--- executor启动task分析

本文基于spark1.6。 上一篇文章提到TaskScheduler将task与executor分配好以后,交给executor来运行task,本文分析executor运行task。首先,在收到消息后...

Spark Task序列化代码分析

Spark的作业会通过DAGScheduler的处理生产许多的Task并构建成DAG图,而分割出的Task最终是需要经过网络分发到不同的Executor。在分发的时候,Task一般都会依赖一些文件和J...
  • LW_GHY
  • LW_GHY
  • 2016年05月23日 11:26
  • 790

Spark中job、stage、task的划分+源码执行过程分析

job、stage、task  Worker Node:物理节点,上面执行executor进程 Executor:Worker Node为某应用启动的一个进程,执行多个tasks Jobs:act...

spark源码学习(七);task任务的提交分析

spark入门学习;task任务的提交分析           spark虽然在计算速度上比hadoop要强势很多,但是这两个框架在底层的数据流都要经过shuffle。由此,shuffle把spark...

Spark技术内幕: Task向Executor提交的源码解析

在上文《Spark技术内幕:Stage划分及提交源码分析》中,我们分析了Stage的生成和提交。但是Stage的提交,只是DAGScheduler完成了对DAG的划分,生成了一个计算拓扑,即需要按照顺...

Spark 任务调度之Executor执行task并返回结果

介绍Executor执行task并返回result给Driver。

Spark使用CombineTextInputFormat缓解小文件过多导致Task数目过多的问题

来自: http://www.cnblogs.com/yurunmiao/p/5195754.html 目前平台使用Kafka + Flume的方式进行实时数据接入,Kafka中的数据由业务方负...

Spark 源码解析:彻底理解TaskScheduler的任务提交和task最佳位置算法

上篇文章《  Spark 源码解析 : DAGScheduler中的DAG划分与提交 》介绍了DAGScheduler的Stage划分算法。 原创文章,转载请注明:转载自 听风居士博客(...

spark集群环境下Lost task 0.0 in stage 10.0 (TID 17, 10.28.23.202): java.io.FileNotFoundException

spark从当前目录加载文件报错,Lost task 0.0 in stage 10.0 (TID 17, 10.28.23.202): java.io.FileNotFoundException,明...
内容举报
返回顶部
收藏助手
不良信息举报
您举报文章:spark task启动
举报原因:
原因补充:

(最多只允许输入30个字)