spark 节点交互

m0_37294549

已于 2022-02-08 00:10:01 修改

阅读量824

点赞数

文章标签： spark 交互大数据

于 2022-02-06 11:51:03 首次发布

本文链接：https://blog.csdn.net/m0_37294549/article/details/122796762

版权

在这里插入图片描述
spark 各计算Endpoint 节点rpc消息和节点间的消息交互

master::RpcMessage

RequestSubmitDriver
RequestKillDriver
RequestDriverStatus
DriverStateChanged

RequestExecutors
KillExecutors

RequestMasterState

ElectedLeader
CompleteRecovery

worker:: OneWayMessage

MasterChanged

LaunchDriver
driverStateChanged
KillDriver

LaunchExecutor
executorStateChanged
ReconnectWorker
KillExecutor

worker:: RpcMessage

RequestWorkerState:WebUI返回worker状态

CoarseGrainedSchedulerBackend:OneWayMessage

ReviveOffers
KillTask

CoarseGrainedExecutorBackend:OneWayMessage

RegisteredExecutor

LaunchTask
KillTask

StopExecutor
Shutdown

消息交互

client $baseContext/create/ --> master::RequestSubmitDriver --> worker::LaunchDriver + worker::LaunchExecutor

client $baseContext/kill/ --> master::RequestKillDriver --> worker::KillDriver

DAGScheduler.submitMissingTasks --> TaskScheduler.submitTasks --> CoarseGrainedSchedulerBackend::ReviveOffers --> CoarseGrainedExecutorBackend::LaunchTask

driverRunner 执行完后， 发送DriverStateChanged --> worker, worker 发给master， 如果失败master schedule 重新调度driver

master.ZooKeeperLeaderElectionAgent.electedLeader （PersistenceEngine恢复driver, worker, appInfo ）-->  	 worker::MasterChanged

master::CompleteRecovery master重新选举后， 重新调度schedule driver, workers