基于上文(spark-core_23: TaskSchedulerImpl.start()、SparkDeploySchedulerBackend.start()、CoarseGrainedExecutorBackend.start()、AppClient.start()源码分析)
11,AppClient初始化ClientEndpoint初始化时调用tryRegisterAllMasters,触发
masterRef.send(RegisterApplication(appDescription,self)):查看一下MasterRpcEndPoint,接收到RegisterApplication这个case class做了哪些工作
//receive方法接收EndpointRef send方法发送的信息
override def receive: PartialFunction[Any, Unit] ={
。。。为了分析流程,将不相关的代码去掉
//注册appliction,是在SparkContext初始化时启动TaskSchedulerImpl.start()之后由SparkDeploySchedulerBackend的AppClient的ClientEndpoint调用的
//这里的description包含了app的具体信息,包括command信息,里面有启动类CoarseGrainedExecutorBackend;这里的driver是ClientEndpoint本身
case RegisterApplication(description, driver) => {
// TODOPrevent repeated registrations from some driver
if (state == RecoveryState.STANDBY) {
// ignore, don't send response
} else{
//description.name就 是sparkconf.setAppName的值
logInfo("Registeringapp " + description.name)
//driver:NettyRpcEndpointRef(spark://AppClient@192.168.1.152:60112)
//app: ApplicationInfo里面包括command信息,里面有启动类CoarseGrainedExecutorBackend,appId的信息(app-20180404172558-0000)
val app= createApplication(description, driver)
//将ApplicationInfo存放到master中有关操作ApplicationInfo的成员变量中
registerApplication(app)
logInfo("Registered app " + description.name + " with ID " + app.id)
//使用ZooKeeperPersistenceEngine,将信息存储到zk中
persistenceEngine.addApplication(app)
//ClientEndpoint发送RegisteredApplication(app.id, MasterRpcEndpointRef)
driver.send(RegisteredApplication(app.id, self))
schedule()
}
}
12,先回到AppClient内部类ClientEndpoint,因为master调用driver.send(RegisteredApplication(app.id, self))
override def receive: PartialFunction[Any, Unit] ={
//RegisteredApplication(app-20180404172558-0000,masterRpcEndpointRef)
case RegisteredApplication(appId_, masterRef) =>
// FIXMEHow to handle the following cases?
// 1. A master receives multiple registrations and sendsback multiple
// RegisteredApplications due to anunstable network.
// 2. Receive multipleRegisteredApplication from different masters because the master is changing.
/**如何处理下面两种情况:
* 1,master接收多次注册和返回多次RegisteredApplication给AppClient,因为不稳定的网络造成的
* 2,AppClient接收多个RegisteredApplication,从不同的master过来的,因为master改变了 */
//appId: AtomicReference[String] ;registered:AtomicBoolean(false)默认是false,注册后设置true
appId.set(appId_)
registered.set(true)
//master:Option[RpcEndpointRef],将masterRpcEndpointRef引用存放到成员变量中
master = Some(masterRef)
//listener就是SparkDeploySchedulerBackend
//让Semaphore(0)信号量释放一个许可,让阻塞在registrationBarrier.acquire()的线程通过
//就是让SparkDeploySchedulerBackend.start()方法中阻塞在waitForRegistration()的线程通过
listener.connected(appId.get)
。。。。
===》进入SparkDeploySchedulerBackend的connected方法
/** 该方法会在AppClient.start之后,向master注册RegisterApplication,注册进去之后,会回复RegisteredApplication给ClientEndpoint
* 在receive里面会调用这个connected
* 参数appId: app-20180404172558-0000
*/
override def connected(appId: String) {
logInfo("Connected to Spark cluster with app ID " + appId)
this.appId = appId
//registrationBarrier:Semaphore(0),通知信号量释放一个许可,让阻塞在registrationBarrier.acquire()的线程通过
//找到的一下只有SparkDeploySchedulerBackend.start()方法调用waitForRegistration()
notifyContext()
//默认不会联接SocketServer
launcherBackend.setAppId(appId)
}
===》进入notifyContext()方法
private def notifyContext() = {
//registrationBarrier:Semaphore(0),通知信号量释放一个许可,让阻塞在registrationBarrier.acquire()的线程通过
//找到的一下只有SparkDeploySchedulerBackend.start()方法调用waitForRegistration()
registrationBarrier.release()
}
===》查看一下只有阻塞在waitForRegistration()的线程
override def start() {
super.start()
launcherBackend.connect()
。。。。
client = new AppClient(sc.env.rpcEnv, masters, appDesc, this, conf)
client.start()
//默认情况下launcherBackend不起作用
launcherBackend.setState(SparkAppHandle.State.SUBMITTED)
//只有AppClient的ClientEndpoint向master注册RegisterApplication成功之后,master发RegisteredApplication给ClientEndpoint,阻塞在waitForRegistration()线程才能通过
waitForRegistration()
launcherBackend.setState(SparkAppHandle.State.RUNNING)
}
13,master发向ClientEndPoint发送RegisteredApplication之后,自己调用了一下scheduler()
override def receive: PartialFunction[Any, Unit] ={
。。。为了分析流程,将不相关的代码省略了
case RegisterApplication(description, driver) => {
….
//ClientEndpoint发送RegisteredApplication(app.id,MasterRpcEndpointRef)
driver.send(RegisteredApplication(app.id, self))
schedule()
}
}
14,scheduler()会排可用的资源给等待的apps,每次有应用程序加入或资源变得可用时都会调用这个方法
/**
* Schedule the currently availableresources among waiting apps. This method will be called
* every time a new app joins or resourceavailability changes.
*/
private def schedule(): Unit= {
if (state != RecoveryState.ALIVE) { return }
// Drivers take strict precedence over executors
//会将workers:HashSet[WorkerInfo],进行重新调整顺序,
// 这个Worker集合是在Worker启动之后向master注册RegisterWorker,然后调registerWorker()加进去的
val shuffledWorkers= Random.shuffle(workers) //Randomization helps balance drivers
//new WorkerInfo().state的值是WorkerState.ALIVE,waitingDrivers集合是对应Cluster模式的,client模式是没有元素的
//WorkerInfo里面的成员值如:workerId: worker-20180321165947-luyl153-workrpc的port值, host:work的host
//port = workrpc.address.port,workerRpcEndPoint , publicAddress: 当前worker的主机名
for (worker<- shuffledWorkers if worker.state ==WorkerState.ALIVE) {
//waitingDrivers:ArrayBuffer[DriverInfo], DriverInfo针对StandaloneRestServer,即cluster模式,StandaloneRestServer是给RestSubmissionClient
//当前分析的是client模式,所以for不执行
for (driver<- waitingDrivers) {
if (worker.memoryFree>= driver.desc.mem && worker.coresFree >= driver.desc.cores) {
launchDriver(worker, driver)
waitingDrivers -= driver
}
}
}
//会让worker启动CoarseGrainedExecutorBackend
startExecutorsOnWorkers()
}
15、查看startExecutorsOnWorkers():分析每个worker是否有足够的core和memery来启动CoarseGrainedExecutorBackend进程
/**
* Schedule and launch executors onworkers
* 安排启动executor在worker节点上,executor就是CoarseGrainedExecutorBackend进程
*/
private def startExecutorsOnWorkers(): Unit = {
// Right now this is a very simple FIFO scheduler. Wekeep trying to fit in the first app
// in the queue, then the second app,etc.
//1,刚开始启动start-all.sh时,waitingApps集合是没有元素的
//2,waitingApps集合元素是在SparkContext初始化时启动TaskSchedulerImpl.start()之后由SparkDeploySchedulerBackend的AppClient的RpcEndPonit调用registerApplication 放进去的
//app:ApplicationInfo里面包括command信息,里面有启动类CoarseGrainedExecutorBackend,appId的信息(app-20180404172558-0000)
for (app<- waitingApps if app.coresLeft > 0) {
//desc:ApplicationDescription的coresPerExecutor就是--executor-cores的值或spark.executor.cores的值,没有设置就是None
val coresPerExecutor:Option[Int] = app.desc.coresPerExecutor
// Filter out workers that don't have enough resources tolaunch an executor
//过滤掉没有足够的资源来启动executor即CoarseGrainedExecutorBackend的worker节点。
//app.desc.memoryPerExecutorMB:它的值对应sc.executorMemory默认是1024MB
// worker.memoryFree:默认给服务器留1g,其它都拿来使用
// app.desc.memoryPerExecutorMB:默认是1024MB对应sc.executorMemory
//worker.coresFree:默认等于worker节点的cpu最大值
val usableWorkers= workers.toArray.filter(_.state ==WorkerState.ALIVE)
.filter(worker =>worker.memoryFree >= app.desc.memoryPerExecutorMB &&worker.coresFree >= coresPerExecutor.getOrElse(1))
.sortBy(_.coresFree).reverse
/** app: ApplicationInfo里面包括command信息,里面有启动类CoarseGrainedExecutorBackend,appId的信息(app-20180404172558-0000)
* spreadOutApps:spark.deploy.spreadOut默认是true
*
返回的数组:在--num-executors或SparkConf的"spark.executor.cores"设置了值情况下对应minCoresPerExecutor变量值
如果不设置值,一个worker节点只会启动一个CoarseGrainedExecutorBackend,只是一个CoarseGrainedExecutorBackend进程会有多个core而以
如果设置了值:返回的数组的索引对应可用usableWorkers数组,work加一个CoarseGrainedExecutorBackend,索引上元素值+=minCoresPerExecutor的值,
worker如果有n个CoarseGrainedExecutorBackend,索引对应的值就是n*minCoresPerExecutor个数的core
*/
val assignedCores = scheduleExecutorsOnWorkers(app, usableWorkers, spreadOutApps)
16,查看一下scheduleExecutorsOnWorkers()是按什么给worker分配一个CoarseGrainedExecutorBackend还是多个CoarseGrainedExecutorBackend进程的
* 安排CoarseGrainedExecutorBackend在worker启动,返回一个array包含一定数量的cores给每个worker
*
* 有两种模式来启动CoarseGrainedExecutorBackend,
* 第一种,尽可能在每个worker节点上进行扩散(默认,基于数据本地性考虑的);
* 第二种则是启动部分worker。
*
* 每个Executor即CoarseGrainedExecutorBackend对应的core的数量是可以配制的通过--num-executors或SparkConf的"spark.executor.cores"进行显示的设置
* 如果Worker有足够的cores或内存,可能一个worker会启动多个Executor即CoarseGrainedExecutorBackend,
* 否则,每个CoarseGrainedExecutorBackend会得到worker中默认的可用的cores数量,在这种情况下一个worker只会启动一个CoarseGrainedExecutorBackend
*
*分配coresPerExecutor即--num-executors或SparkConf的"spark.executor.cores",是很重要的,而不是采用默认情况下不设置值,哪spark.executor.cores的值就是1
* 如:有4个节点,每个节点16个core,一共有64个core,如果用户要启动3个Executor即CoarseGrainedExecutorBackend,
* 要设置每个worker:spark.executor.cores是16个(值是几个就会有几个CoarseGrainedExecutorBackend),总共spark.cores.max是48
* 而不是按默认情况一次一个core
*
* //app: ApplicationInfo里面包括command信息,里面有启动类CoarseGrainedExecutorBackend,appId的信息(app-20180404172558-0000)
* // spreadOutApps是boolean默认是true
*
* 为Application分配资源选择Worker(Executor),现在有两种策略:
* 1.尽量打散
* 将一个Application尽可能的分配到不同的节点,可以通过设置spark.deploy.spreadOut来实现对应方法实参spreadOutApps,默认值为true
* 2.尽量集中
* 将一个Application金肯呢个分配到尽可能少的节点,CPU密集型而内存占用比较少的Application比较适合这种策略
*/
private def scheduleExecutorsOnWorkers(
app: ApplicationInfo,
usableWorkers: Array[WorkerInfo],
spreadOutApps: Boolean): Array[Int] = {
//如果--num-executors或SparkConf的"spark.executor.cores"不设置每个worker上的coresPerExecutor的值是None
val coresPerExecutor= app.desc.coresPerExecutor
//如果不给--num-executors或SparkConf的"spark.executor.cores"设置值,minCoresPerExecutor的默认值是1,每个worker只会启动一个CoarseGrainedExecutorBackend,
val minCoresPerExecutor= coresPerExecutor.getOrElse(1)
//不设置--num-executors或SparkConf的"spark.executor.cores",这个oneExecutorPerWorker的值就是true
val oneExecutorPerWorker= coresPerExecutor.isEmpty
//memoryPerExecutor值来至sc.executorMemory,默认是1024MB
val memoryPerExecutor= app.desc.memoryPerExecutorMB
//得到可用的Worker个数
val numUsable= usableWorkers.length
//按可用的worker长度来设置assignedCores数组
val assignedCores= new Array[Int](numUsable) // Number of cores to give to each worker。一定数量的cores给每个worker
val assignedExecutors= new Array[Int](numUsable) // Number of new executors on each worker.一定数量的CoarseGrainedExecutorBackend给每个worker
//coresLeft的默认值:Int.MaxValue,coresFree默认等于worker节点的cpu最大值 - 0,将所有可用的worker的cpu加起来
//coresToAssign的值应该是所有worker的cpu之和
var coresToAssign= math.min(app.coresLeft, usableWorkers.map(_.coresFree).sum)
/** Return whether the specified worker can launch anexecutor for this app.
* 返回指定worker是否可以启动CoarseGrainedExecutorBackend给这个app。*/
def canLaunchExecutor(pos: Int): Boolean = {
//coresToAssign的值应是所有worker的cpu之和,
// 如果不给--num-executors或SparkConf的"spark.executor.cores"设置值,minCoresPerExecutor的值是1
val keepScheduling= coresToAssign >= minCoresPerExecutor
//coresFree默认等于worker节点的cpu最大值;minCoresPerExecutor的默认值是1
// assignedCores:new Array[Int](numUsable)如果第一次进来,只是声明了数组的长度,所以每个元素的默认值是0
val enoughCores= usableWorkers(pos).coresFree - assignedCores(pos) >= minCoresPerExecutor
// If we allow multiple executors per worker, then we canalways launch new executors.
// Otherwise, if there is alreadyan executor on this worker, just give it more cores.
//如果允许在每个worker上启动多个CoarseGrainedExecutorBackend,这样每次启动新的CoarseGrainedExecutorBackend。
//否则,已经在已有CoarseGrainedExecutorBackend增加更多的cores
/**
* 不设置--num-executors或SparkConf的"spark.executor.cores",这个oneExecutorPerWorker的值就是true,
* 否则为false,这样就可以一直启动新的CoarseGrainedExecutorBackend
* assignedExecutors:如果第一次进来,只是声明了数组的长度,所以每个元素的默认值是0
* 这样launchingNewExecutor的值就是true
*/
val launchingNewExecutor = !oneExecutorPerWorker ||assignedExecutors(pos) == 0
if (launchingNewExecutor){
//第一次:0* memoryPerExecutor:1024 = 0
val assignedMemory = assignedExecutors(pos) *memoryPerExecutor
//coresFree默认等于worker节点的cpu最大值 - 0
val enoughMemory = usableWorkers(pos).memoryFree -assignedMemory >= memoryPerExecutor
//第一次app.executors的默认值new mutable.HashMap[Int, ExecutorDesc],
// app.executorLimit的默认值是Integer.MAX_VALUE
val underLimit = assignedExecutors.sum + app.executors.size < app.executorLimit
//在这四个变量都为true时,canLaunchExecutor方法参数上索引对应的usableWorkers数组上的worker表示可运行CoarseGrainedExecutorBackend
keepScheduling && enoughCores &&enoughMemory && underLimit
} else {
// We're adding cores to an existing executor, so no needto check memory and executor limits
//我们将增加core到已存在的CoarseGrainedExecutorBackend,所以没有必有检查内存和CoarseGrainedExecutorBackend的限制
keepScheduling && enoughCores
}
}
// Keep launching executors until no more workers canaccommodate any more executors, or if wehave reached this application's limits
//一直启动CoarseGrainedExecutorBackend直到worker不能来容纳更多的CoarseGrainedExecutorBackend,或达到应用的限制
//scala> (0 until3).filter(_!=1).foreach(println)
//打印:0 2,所以会将不符合条件的索引去掉
var freeWorkers= (0 until numUsable).filter(canLaunchExecutor)
while (freeWorkers.nonEmpty){
freeWorkers.foreach { pos =>
var keepScheduling= true
while (keepScheduling &&canLaunchExecutor(pos)) {
//如果不给--num-executors或SparkConf的"spark.executor.cores"设置值,minCoresPerExecutor的默认值是1
//coresToAssign的值应该是所有worker的cpu之和,每次自减minCoresPerExecutor
coresToAssign -= minCoresPerExecutor
//同时给assignedCores= new Array[Int](numUsable)对应索引上元素值,每次自增minCoresPerExecutor
assignedCores(pos) += minCoresPerExecutor
// If we are launching one executor per worker, thenevery iteration assigns 1 core to theexecutor. Otherwise, every iteration assigns cores to a new executor.
//如果我们将在每个worker上启动一个CoarseGrainedExecutorBackend,后面每次遍历都会赋一个core给CoarseGrainedExecutorBackend
//否则,每次遍历会给新的CoarseGrainedExecutorBackend赋core
/**
* 从这个代码可以很清楚的看出,给--num-executors或SparkConf的"spark.executor.cores"设置值,可以用多个CoarseGrainedExecutorBackend来执行任务
*/
if (oneExecutorPerWorker) {
assignedExecutors(pos) = 1
} else{
//assignedExecutors = new Array[Int](numUsable),这个数组决定每个Worker,能启动多少个CoarseGrainedExecutorBackend
assignedExecutors(pos) += 1
}
// Spreading out an application means spreading out itsexecutors across as many workers aspossible.
// If we are not spreading out,then we should keep scheduling executors on this worker until we use all of itsresources.
// Otherwise, just move on tothe next worker.
/**
* spreadOutApps的值是master初始化时设置的对应:apache.spark.deploy.spreadOut,默认值为true
* 代码也证明,注释说的
* 有两种模式来启动CoarseGrainedExecutorBackend,
* 第一种,尽可能在每个worker节点上进行扩散(默认,基于数据本地性考虑的);
* 第二种,如果spreadOut的值是false,则会安排多个CoarseGrainedExecutorBackend在一个worker上直到把所有资源都用完。
*/
if (spreadOutApps) {
keepScheduling = false //false会跳出循环
}
}
}
//在上面freeWorkers.foreach之后,再次过滤一下最开始的可用的worker是否还能再启动CoarseGrainedExecutorBackend
//即freeWorkers回到while (freeWorkers.nonEmpty),是否还能再次进入给assignedExecutors数组赋值(代码值得学习)
//assignedExecutors = newArray[Int](numUsable),这个数组决定每个Worker,能启动多少个CoarseGrainedExecutorBackend
freeWorkers = freeWorkers.filter(canLaunchExecutor)
}
//assignedCores = new Array[Int](numUsable)每增加一个CoarseGrainedExecutorBackend就会给该数组中索引上的元素加minCoresPerExecutor
// 返回的数组:在--num-executors或SparkConf的"spark.executor.cores"设置了值情况下对应minCoresPerExecutor变量值
// 如果不设置值,一个worker节点只会启动一个CoarseGrainedExecutorBackend,只是一个CoarseGrainedExecutorBackend进程会有多个core而以
// 如果设置了值:返回的数组的索引对应可用usableWorkers数组,work加一个CoarseGrainedExecutorBackend,索引上元素值+=minCoresPerExecutor的值,
// worker如果有n个CoarseGrainedExecutorBackend,索引对应的值就是n*minCoresPerExecutor个数的core
assignedCores
}
17,再回到startExecutorsOnWorkers()中的allocateWorkerResourceToExecutors方法如何分配资源给CoarseGrainedExecutorBackend
private def startExecutorsOnWorkers(): Unit = {
。。。
val assignedCores= scheduleExecutorsOnWorkers(app, usableWorkers, spreadOutApps)
// Now that we've decided how many cores to allocate oneach worker, let's allocate them
//现在我们要决定好有多少个core可以分配到每个worker上,assignedCores(pos)如果等于0,
// 说明内存或core不符合启动CoarseGrainedExecutorBackend,会将该索引过滤掉
for (pos<- 0 until usableWorkers.length if assignedCores(pos)> 0) {
/**
* 方法中会launchExecutor(worker, exec),仔细看这个launchExecutor(worker: WorkerInfo, exec: ExecutorDesc),它发送了如下两条消息:
*worker.endpoint.send(LaunchExecutor(masterUrl, exec.application.id, exec.id,exec.application.desc, exec.cores, exec.memory))
*exec.application.driver.send(ExecutorAdded(exec.id, worker.id, worker.hostPort,exec.cores, exec.memory))
*
* //app:ApplicationInfo里面包括command信息,里面有启动类CoarseGrainedExecutorBackend,appId的信息(app-20180404172558-0000)
* //coresPerExecutor就是--executor-cores(yarn模式中使用)的值或spark.executor.cores的值,没有设置就是None
*/
allocateWorkerResourceToExecutors(app, assignedCores(pos), coresPerExecutor, usableWorkers(pos))
}
}
}
===》allocateWorkerResourceToExecutors()方法通过第二个参数assignedCores除spark.executor.cores(即第三个参数)的结果就是每个worker对应的Executer
* 分配worker的资源给一个或多个CoarseGrainedExecutorBackend
* app: ApplicationInfo里面包括command信息,里面有启动类CoarseGrainedExecutorBackend,appId的信息(app-20180404172558-0000)
* assignedCores:对应和第四个参数WorkerInfo对应,表示该worker可以有M个core,有多少个CoarseGrainedExecutorBackend等于M 除以 coresPerExecutor
* coresPerExecutor就是--executor-cores的值或spark.executor.cores的值,没有设置就是None,这样一个CoarseGrainedExecutorBackend就会有多个core
*/
private def allocateWorkerResourceToExecutors(
app: ApplicationInfo,
assignedCores: Int,
coresPerExecutor: Option[Int],
worker: WorkerInfo): Unit = {
// If the number of cores per executor is specified, wedivide the cores assigned
// to this worker evenly among the executorswith no remainder.
// Otherwise, we launch a singleexecutor that grabs all the assignedCores on this worker.
//如果每个CoarseGrainedExecutorBackend被指定了--num-executors或SparkConf的"spark.executor.cores"的值,会对该worker总共的assignedCores除coresPerExecutor进行平均分(我没看实现代码,看到下面代码,果真是这样)
//否则会给一个CoarseGrainedExecutorBackend赋assignedCores
//可以看出如果coresPerExecutor不赋值numExecutors就是1
val numExecutors= coresPerExecutor.map { assignedCores / _ }.getOrElse(1)
//Option语法,getOrElse表示如果coresPerExecutor是None则取参数上的值,否则取自己
val coresToAssign= coresPerExecutor.getOrElse(assignedCores)
//to是一个左闭,右闭的集合,也就是有多少个CoarseGrainedExecutorBackend就会循环多少次
for (i<- 1 to numExecutors) {
/**
* 返回ExecutorDesc同时会给ApplicationInfo成员(一个SparkContext对应一个ApplicationInfo): executors: mutable.HashMap[Int, ExecutorDesc],这个id就是executor(CoarseGrainedExecutorBackend)自增数值,
* 就是通过newExecutorId(useID)算的,value就是ExecutorDesc
* coresGranted: //就是对应--num-executors或SparkConf的"spark.executor.cores"的值,每调用一次addExecutor会按cores自增
* (每个worker有多少个CoarseGrainedExecutorBackend就会addExecutor就调多少次)
*
* ExecutorDesc的对象成员
* newExecutorId(useID) 给executor(CoarseGrainedExecutorBackend)赋id值,第一次是0,后面自增1
* app: ApplicationInfo里面包括command信息,里面有启动类CoarseGrainedExecutorBackend,appId的信息(app-20180404172558-0000)
* WorkerInfo:可用来启动的CoarseGrainedExecutorBackend的worker引用
* cores:每个CoarseGrainedExecutorBackend可以有多少个core,如果没有--num-executors或SparkConf的"spark.executor.cores"的值,就只启动一个CoarseGrainedExecutorBackend,把所有可用的core给它
* desc.memoryPerExecutorMB:对应sc.executorMemory,默认是1024MB
*/
val exec = app.addExecutor(worker, coresToAssign)
//是在SparkContext初始化时启动TaskSchedulerImpl.start()之后由SparkDeploySchedulerBackend的AppClient的RpcEndPonit调用registerApplication 放进去的
//再调用startExecutorsOnWorkers==》allocateWorkerResourceToExecutors==》launchExecutor(worker, exec)
launchExecutor(worker, exec)
app.state =ApplicationState.RUNNING
}
}
===》顺便看一下applictionaInfo.addExecutor()实现
private[master] def addExecutor(
worker: WorkerInfo,
cores: Int,
useID: Option[Int] = None):ExecutorDesc = {
/** ExecutorDesc的对象成员
* newExecutorId(useID) 给executor(CoarseGrainedExecutorBackend)赋id值,第一次是0,后面自增1
* app: ApplicationInfo里面包括command信息,里面有启动类CoarseGrainedExecutorBackend,appId的信息(app-20180404172558-0000)
* WorkerInfo:可用来启动的CoarseGrainedExecutorBackend的worker引用
* cores:每个CoarseGrainedExecutorBackend可以有多少个core,如果没有--num-executors(属于yarn的)或SparkConf的"spark.executor.cores"的值,
* 就只启动一个CoarseGrainedExecutorBackend,把所有可用的core给它
* desc.memoryPerExecutorMB:对应sc.executorMemory,默认是1024MB
*/
val exec= new ExecutorDesc(newExecutorId(useID), this, worker, cores, desc.memoryPerExecutorMB)
//
//executors: mutable.HashMap[Int, ExecutorDesc],这个id就是executor(CoarseGrainedExecutorBackend),就是通过newExecutorId(useID)算的
//value就是ExecutorDesc。这两成员都是ApplicationInfo,整个应用只有一个ApplicationInfo
executors(exec.id) = exec
coresGranted += cores //就是对应--num-executors或SparkConf的"spark.executor.cores"的值,每调用一次addExecutor会按cores自增
exec
}
18,再回到allocateWorkerResourceToExecutors方法调用lauchExecutor(),查看master让各个Worker启动CoarseGrainedExecutorBackend
private def allocateWorkerResourceToExecutors(
app: ApplicationInfo,
assignedCores: Int,
coresPerExecutor: Option[Int],
worker: WorkerInfo): Unit = {
…
for (i<- 1 to numExecutors) {
…
launchExecutor(worker, exec)
app.state =ApplicationState.RUNNING
}
}
private def launchExecutor(worker: WorkerInfo, exec:ExecutorDesc): Unit = {
//Launching executor app-20180508232008-0000/5 on workerworker-20180508231819-192.168.1.155-33242
logInfo("Launchingexecutor " + exec.fullId + " on worker " + worker.id)
/** 一个worker有多少个CoarseGrainedExecutorBackend,该方法就会被执行几次
* 方法会给WorkerInfo赋如下值:
* //executors:HashMap[String, ExecutorDesc]它的key是: app-20180404172558-0000/0,对应的值是这个id对应的ExecutorDesc
* //coresUsed: 每个CoarseGrainedExecutorBackend可以有多少个core,如果没有--num-executors或SparkConf的"spark.executor.cores"的值,
就只启动一个CoarseGrainedExecutorBackend,把所有可用的core给它
//memoryUsed: 对应sc.executorMemory,默认是1024MB
*/
worker.addExecutor(exec)
….}
===>先看一下每启动一个CoarseGrainedExecutorBackend时,对WorkerInfo的成员的操作
/**
* 该方法是master的RegisterApplication被触发==》通过master启动CoarseGrainExecutorBackend的allocateWorkerResourceToExecutors==》launchExecutor()调用的一个worker有多少个CoarseGrainedExecutorBackend,该方法就会被执行几次
*/
def addExecutor(exec:ExecutorDesc) {
//executors:HashMap[String, ExecutorDesc]它的key是: app-20180404172558-0000/0,对应的值是这个id对应的ExecutorDesc
executors(exec.fullId) = exec
//coresUsed: 每个CoarseGrainedExecutorBackend可以有多少个core,如果没有--num-executors(yarn模式下面用的)或SparkConf的"spark.executor.cores"的值,
//就只启动一个CoarseGrainedExecutorBackend,把所有可用的core给它
coresUsed += exec.cores
//memoryUsed: 对应sc.executorMemory,默认是1024MB
memoryUsed += exec.memory
}
===》再看一下master给Worker发送LaunchExecutor,启动CoarseGrainedExecutorBackend
private def launchExecutor(worker: WorkerInfo, exec:ExecutorDesc): Unit = {
….
/**
* 第一条消息是发给worker让其launchExecutor的,worker本身是个消息通讯体,其在接受到消息LaunchExecutor(masterUrl,appId, execId, appDesc, cores_, memory_) 后,
创建并start了一个(ExecutorRunnerManages the execution of one executor process.)
masterUrl:spark://luyl152:7077
ApplicationInfo.desc,里面包括command信息,里面有启动类CoarseGrainedExecutorBackend
使用Jdk的ProcessBuilder.start()来启动CoarseGrainedExecutorBackend
exec.application.id:app-20180404172558-0000
exec.id: 一个自增的数值,默认从0开始
exec.cores : --num-executors或SparkConf的"spark.executor.cores"的值
exec.memory : 对应sc.executorMemory,默认是1024MB
*/
worker.endpoint.send(LaunchExecutor(masterUrl,
exec.application.id, exec.id, exec.application.desc, exec.cores, exec.memory))
//ExecutorStateChanged消息给master通知Master executor状态的变化 :sendToMaster(ExecutorStateChanged(appId, execId, manager.state,None, None))
exec.application.driver.send(
ExecutorAdded(exec.id, worker.id, worker.hostPort, exec.cores, exec.memory))
/**
* master发送的第二条消息是发送给ClientEndpoint这个消息通讯体通知它获得了executor,ClientEndpoint在接受到
* ExecutorAdded(id: Int, workerId:String, hostPort: String, cores: Int, memory: Int)
* 消息后会调用listener.executorAdded(fullId,workerId, hostPort, cores, memory),
* 实质是调用AppClientListener的具体实现SparkDeploySchedulerBackend.executorAdded(fullId: String, workerId: String,hostPort: String, cores: Int,memory: Int),到此executor注册成功.
*/
}