1、本文目标以及其它说明:
2、executor包下面的数据结构说明
private[spark] trait ExecutorBackend { def statusUpdate(taskId: Long, state: TaskState, data: ByteBuffer): Unit } 用于向driver更新任务的状态
private[spark] class CoarseGrainedExecutorBackend(
override val rpcEnv: RpcEnv, driverUrl: String, executorId: String, hostname: String, cores: Int, userClassPath: Seq[URL], env: SparkEnv) extends ThreadSafeRpcEndpoint with ExecutorBackend with Logging {有一个driver、executor字段,负债和driver通信。负债注册、启动、停止executor,断开和driver的链接。还
负责提交、kill、任务,并将任务状态更新给driver.伴生对象有main方法,这个代表的是一个进程。
private[spark] class CommitDeniedException( msg: String, jobID: Int, splitID: Int, attemptNumber: Int)
extends Exception(msg) {
任务结束后将结果提交到hdfs,但是被driver拒绝了,这个时候会抛这个异常
private[spark] class Executor( executorId: String, executorHostname: String, env: SparkEnv, userClassPath: Seq[URL] = Nil, isLocal: Boolean = false, uncaughtExceptionHandler: UncaughtExceptionHandler = new
SparkUncaughtExceptionHandler) extends Logging {
里面持有了
threadPool字段,(newcachedThreadPool),将任务提交给线程池,杀死任务、关闭线程池。发送心跳信息
其它部分和metrics有关系,暂不予研究