整体流程图
standalone模式简介
standalone模式,是spark自带的资源调度框架,该框架下主要有三个节点:
- client 客户端,将应用程序提交,记录着要运行业务逻辑
- master 负责资源管理
- worker 负责任务的执行
spark Application有个Driver驱动,即用户提交的程序,Driver进程可以运行在Client上也可以运行在worker上 ,其区别如下:
-
client:用户启动Client端,在client端启动Driver进程,在Driver中实例化SparkContext
-
worker: 用户启动Client端,Client提交应用程序给master, master调度资源,指定一个worker节点启动Driver进程
运行流程
sparkContext三大核心:DAGScheduler,TaskScheduler,SchedulerBackend
1)当spark集群启动以后,worker节点会有一个心跳机制和master保持通信;
2)SparkContext连接到master以后会向master申请资源,而master会根据worker心跳来分配worker的资源,并启动worker的executor进程;
3)SparkContext将程序代码解析成DAG,并提交给DagScheduler;
4)DagScheduler根据宽依赖分解成多个stage,每个stage即一个taskSet, 包含着多个task;
5)taskSet会被提交给TaskScheduler,TaskScheduler将task分配到worker,提交给executor进程,executor进程会创建线程池去执行task,并且向SparkContext报告执行情况,直到task完成;
6)所有task完成以后,SparkContext向Master注销并释放资源;
Spark集群启动
启动 Spark 集群,查看 sbin 下的 start-all.sh 脚本,会先启动 Master:
# Start Master
"${SPARK_HOME}/sbin"/start-master.sh
# Start Workers
"${SPARK_HOME}/sbin"/start-slaves.sh
查看 sbin/start-master.sh 脚本,发现会去执行 org.apache.spark.deploy.master.Master 类,开始在源码中跟进Master,从 main 方法开始:
private[deploy] object Master extends Logging {
val SYSTEM_NAME = "sparkMaster"
val ENDPOINT_NAME = "Master"
def main(argStrings: Array[String]) {
Thread.setDefaultUncaughtExceptionHandler(new SparkUncaughtExceptionHandler(
exitOnUncaughtException = false))
Utils.initDaemon(log)
val conf = new SparkConf
val args = new MasterArguments(argStrings, conf)
/**
* 创建RPC环境和Endpoint(RPC 远程过程调用),在Spark中Master和Worker角色都有各自的Endpoint,相当于各自的通信邮箱。
*/
val (rpcEnv, _, _) = startRpcEnvAndEndpoint(args.host, args.port, args.webUiPort, conf)
rpcEnv.awaitTermination()
}
在 main 方法中执行了 startRpcEnvAndEndpoint 方法,创建RpcEnv和 Endpoint,RpcEnv 是用于接收消息和处理消息的远程通信环境,Master 向 RpcEnv 中去注册,不管是 Master,Driver,Worker,Executor 都有自己的 Endpoint,相当于是邮箱,大意是节点之间的通信首先需要获取到目的节点的邮箱。Master启动时会将 Endpoint 注册在 RpcEnv 里面,用于接收、处理消息。
跟进 startRpcEnvAndEndpoint 方法:
/**
* Start the Master and return a three tuple of:
* (1) The Master RpcEnv
* (2) The web UI bound port
* (3) The REST server bound port, if any
*/
def startRpcEnvAndEndpoint(
host: String,
port: Int,
webUiPort: Int,
conf: SparkConf): (RpcEnv, Int, Option[Int]) = {
val securityMgr = new SecurityManager(conf)
/**
* 创建RpcEnv,后续向RpcEnv中注册[Driver,Master,Worker,Executor]
*/
val rpcEnv: RpcEnv = RpcEnv.create(SYSTEM_NAME, host, port, conf, securityMgr)
/**
* 向RpcEnv中注册Master
*
* rpcEnv.setupEndpoint(name,new Master)
* 新建Master伴生类对象,继承ThreadSafeRpcEndpoint,归根结底继承Trait RpcEndpoint
* EndPoint中存在
* onstart() :启动当前Endpoint
* receive() :负责收消息
* receiveAndReply():接受消息并回复
* Endpoint 还有各自的引用,方便其他Endpoint发送消息,直接引用对方的EndpointRef 即可找到对方的Endpoint
*
* 以下 masterEndpoint就是Master的Endpoint引用RpcEndpointRef
* RpcEndpointRef中存在:
* send():发送消息
* ask() :请求消息,并等待回应。
*/
val masterEndpoint = rpcEnv.setupEndpoint(ENDPOINT_NAME,
new Master(rpcEnv, rpcEnv.address, webUiPort, securityMgr, conf))
val portsResponse = masterEndpoint.askSync[BoundPortsResponse](BoundPortsRequest)
(rpcEnv, portsResponse.webUIPort, portsResponse.restPort)
继续跟进 create 方法,查看创建RpcEnv 的原理:
create(name, host, host, port, conf, securityManager, 0, clientMode)
继续跟进 create 方法:
// 配置RPC环境
val config = RpcEnvConfig(conf, name, bindAddress, advertiseAddress, port, securityManager,
numUsableCores, clientMode)
// 工厂模式创建NettyRpcEnv
new NettyRpcEnvFactory().create(config)
创建一个 NettyRpcEnvFactory 的对象,并调用 create 方法,跟进此方法:
// 实例化NettyRpcEnv,后续返回
val nettyEnv =
new NettyRpcEnv(sparkConf, javaSerializerInstance, config.advertiseAddress,
config.securityManager, config.numUsableCores)
if (!config.clientMode) {
val startNettyRpcEnv: Int => (NettyRpcEnv, Int) = { actualPort =>
nettyEnv.startServer(config.bindAddress, actualPort)
(nettyEnv, nettyEnv.address.port)
}
实例化过程中会进行一些初始化操作:
private val dispatcher: Dispatcher = new Dispatcher(this, numUsableCores)
private val streamManager = new NettyStreamManager(this)
private val transportContext = new TransportContext(transportConf,
new NettyRpcHandler(dispatcher, this, streamManager))
- Dispatcher:存放消息的队列和消息的转发
- TransportContext:可以创建NettyRpcHandler
Dispatcher
首先看 Dispatcher 实例化时执行的逻辑:
/** Thread pool used for dispatching messages. */
private val threadpool: ThreadPoolExecutor = {
val availableCores =
if (numUsableCores > 0) numUsableCores else Runtime.getRuntime.availableProcessors()
val numThreads = nettyEnv.conf.getInt("spark.rpc.netty.dispatcher.numThreads",
math.max(2, availableCores))
val pool = ThreadUtils.newDaemonFixedThreadPool(numThreads, "dispatcher-event-loop")
for (i <- 0 until numThreads) {
pool.execute(new MessageLoop)
}
pool
}
在 Dispatcher 实例化的过程中会创建一个 threadpool,在 threadpool 中会执行 MessageLoop:
/** Message loop used for dispatching messages. */
private class MessageLoop extends Runnable {
override def run(): Unit = {
try {
while (true) {
try {
val data = receivers.take()
if (data == PoisonPill) {
// Put PoisonPill back so that other MessageLoops can see it.
receivers.offer(PoisonPill)
return
}
data.inbox.process(Dispatcher.this)
} catch {
case NonFatal(e) => logError(e.getMessage, e)
}
}
} catch {
case _: InterruptedException => // exit
case t: Throwable =>
try {
// Re-submit a MessageLoop so that Dispatcher will still work if
// UncaughtExceptionHandler decides to not kill JVM.
threadpool.execute(new MessageLoop)
} finally {
throw t
}
}
}
/** Message loop used for dispatching messages. */
private class MessageLoop extends Runnable {
override def run(): Unit = {
try {
while (true) {
try {
val data = receivers.take()
if (data == PoisonPill) {
// Put PoisonPill back so that other MessageLoops can see it.
receivers.offer(PoisonPill)
return
}
data.inbox.process(Dispatcher.this)
在 MessageLoop 中的 receivers.take() 会一直向 receivers消息队列中取数据data,然后调用data.inbox.process(Dispatcher.this);
而 receivers 是在 Dispatcher 初始化时创建的,至此接收消息的程序已经启动起来。
// Track the receivers whose inboxes may contain messages.
private val receivers = new LinkedBlockingQueue[EndpointData]
其中传入的 EndpointData 对象,实例化时会实例化一个 Inbox 对象:
private class EndpointData(
val name: String,
val endpoint: RpcEndpoint,
val ref: NettyRpcEndpointRef) {
val inbox = new Inbox(ref, endpoint)
}
实例化 Inbox 对象时初始化message,随即加入OnStart样例类
protected val messages = new java.util.LinkedList[InboxMessage]()
inbox.synchronized {
messages.add(OnStart)
}
data.inbox.process(Dispatcher.this)方法的调用:
while (true) {
safelyCall(endpoint) {
message match {
case RpcMessage(_sender, content, context) =>
try {
endpoint.receiveAndReply(context).applyOrElse[Any, Unit](content, { msg =>
throw new SparkException(s"Unsupported message $message from ${_sender}")
})
} catch {
case e: Throwable =>
context.sendFailure(e)
// Throw the exception -- this exception will be caught by the safelyCall function.
// The endpoint's onError function will be called.
throw e
}
case OneWayMessage(_sender, content) =>
endpoint.receive.applyOrElse[Any, Unit](content, { msg =>
throw new SparkException(s"Unsupported message $message from ${_sender}")
})
case OnStart =>
endpoint.onStart()
若消息类型为OnStart时,调用 endpoint.onStart()
TransportContext
NettyRpcEnv实例化先告一段落,回到NettyRpcEnvFactory的create方法继续往下执行
if (!config.clientMode) {
val startNettyRpcEnv: Int => (NettyRpcEnv, Int) = { actualPort =>
nettyEnv.startServer(config.bindAddress, actualPort)
(nettyEnv, nettyEnv.address.port)
}
跟进startServer方法:
def startServer(bindAddress: String, port: Int): Unit = {
val bootstraps: java.util.List[TransportServerBootstrap] =
if (securityManager.isAuthenticationEnabled()) {
java.util.Arrays.asList(new AuthServerBootstrap(transportConf, securityManager))
} else {
java.util.Collections.emptyList()
}
server = transportContext.createServer(bindAddress, port, bootstraps)
dispatcher.registerRpcEndpoint(
RpcEndpointVerifier.NAME, new RpcEndpointVerifier(this, dispatcher))
}
transportContext.createServer 方法是用来启动 NettyRpc 服务的,方法中实例化 TransportServer对象
/** Create a server which will attempt to bind to a specific host and port. */
public TransportServer createServer(
String host, int port, List<TransportServerBootstrap> bootstraps) {
return new TransportServer(this, host, port, rpcHandler, bootstraps);
}
TransportServer构造函数中,在 try 中会调用 init 方法进行初始化:
boolean shouldClose = true;
try {
init(hostToBind, portToBind);
shouldClose = false;
} finally {
if (shouldClose) {
JavaUtils.closeQuietly(this);
}
}
在 init 方法中初始化netty中的bossGroup和workerGroup,通过bootstrap引导类启动netty Rpc服务, 的远程通信对象 bootstrap 会调用 childHandler 方法,会初始化网络通信管道:
private void init(String hostToBind, int portToBind) {
IOMode ioMode = IOMode.valueOf(conf.ioMode());
// netty中的bossGroup
EventLoopGroup bossGroup = NettyUtils.createEventLoop(ioMode, 1,
conf.getModuleName() + "-boss");
// netty中的workerGroup
EventLoopGroup workerGroup = NettyUtils.createEventLoop(ioMode, conf.serverThreads(),
conf.getModuleName() + "-server");
PooledByteBufAllocator allocator = NettyUtils.createPooledByteBufAllocator(
conf.preferDirectBufs(), true /* allowCache */, conf.serverThreads());
// netty中的bootstrap
bootstrap = new ServerBootstrap()
.group(bossGroup, workerGroup)
.channel(NettyUtils.getServerChannelClass(ioMode))
.option(ChannelOption.ALLOCATOR, allocator)
.option(ChannelOption.SO_REUSEADDR, !SystemUtils.IS_OS_WINDOWS)
.childOption(ChannelOption.ALLOCATOR, allocator);
this.metrics = new NettyMemoryMetrics(
allocator, conf.getModuleName() + "-server", conf);
if (conf.backLog() > 0) {
bootstrap.option(ChannelOption.SO_BACKLOG, conf.backLog());
}
if (conf.receiveBuf() > 0) {
bootstrap.childOption(ChannelOption.SO_RCVBUF, conf.receiveBuf());
}
if (conf.sendBuf() > 0) {
bootstrap.childOption(ChannelOption.SO_SNDBUF, conf.sendBuf());
}
// 设置childHandler
bootstrap.childHandler(new ChannelInitializer<SocketChannel>() {
@Override
protected void initChannel(SocketChannel ch) {
RpcHandler rpcHandler = appRpcHandler;
for (TransportServerBootstrap bootstrap : bootstraps) {
rpcHandler = bootstrap.doBootstrap(ch, rpcHandler);
}
context.initializePipeline(ch, rpcHandler);
}
});
InetSocketAddress address = hostToBind == null ?
new InetSocketAddress(portToBind): new InetSocketAddress(hostToBind, portToBind);
// 绑定地址
channelFuture = bootstrap.bind(address);
channelFuture.syncUninterruptibly();
port = ((InetSocketAddress) channelFuture.channel().localAddress()).getPort();
logger.debug("Shuffle server started on port: {}", port);
}
bootstrap 会调用 childHandler 方法,会初始化网络通信管道:
// 初始化网络通信管道
context.initializePipeline(ch, rpcHandler);
public TransportChannelHandler initializePipeline(
SocketChannel channel,
RpcHandler channelRpcHandler) {
try {
// 自定义一个ChannelHandler
TransportChannelHandler channelHandler = createChannelHandler(channel, channelRpcHandler);
// 将需要的Handler添加到pipeline中
channel.pipeline()
.addLast("encoder", ENCODER)
.addLast(TransportFrameDecoder.HANDLER_NAME, NettyUtils.createFrameDecoder())
.addLast("decoder", DECODER)
.addLast("idleStateHandler", new IdleStateHandler(0, 0, conf.connectionTimeoutMs() / 1000))
// NOTE: Chunks are currently guaranteed to be returned in the order of request, but this
// would require more logic to guarantee if this were not part of the same event loop.
.addLast("handler", channelHandler);
return channelHandler;
} catch (RuntimeException e) {
logger.error("Error while initializing Netty pipeline", e);
throw e;
}
}
在初始化网络通信管道的过程中,创建处理消息的 channelHandler 对象,该对象的作用是
- 创建并处理客户端的请求消息和服务消息
/**
* Creates the server- and client-side handler which is used to handle both RequestMessages and
* ResponseMessages. The channel is expected to have been successfully created, though certain
* properties (such as the remoteAddress()) may not be available yet.
*/
private TransportChannelHandler createChannelHandler(Channel channel, RpcHandler rpcHandler) {
TransportResponseHandler responseHandler = new TransportResponseHandler(channel);
TransportClient client = new TransportClient(channel, responseHandler);
TransportRequestHandler requestHandler = new TransportRequestHandler(channel, client,
rpcHandler, conf.maxChunksBeingTransferred());
return new TransportChannelHandler(client, responseHandler, requestHandler,
conf.connectionTimeoutMs(), closeIdleConnections);
}
TransportChannelHandler 由以上 responseHandler,client,requestHandler 三个 handler 构建,并且这个对象中有 channelRead 方法,用于读取接收到的消息:
@Override
public void channelRead(ChannelHandlerContext ctx, Object request) throws Exception {
// 判断当前消息的类型,是请求消息还是响应消息
if (request instanceof RequestMessage) {
requestHandler.handle((RequestMessage) request);
} else if (request instanceof ResponseMessage) {
responseHandler.handle((ResponseMessage) request);
} else {
ctx.fireChannelRead(request);
}
}
以 requestHandler 为例,调用 handle —> processRpcRequest((RpcRequest) request),会看到 rpcHandler.receive,此时调用的是 NettyRpcHandler 的 receive:
private void processRpcRequest(final RpcRequest req) {
// rpcHandler是传过来的NettyRpcHandler,所以调用NettyRpcHandler.receive
try {
rpcHandler.receive(reverseClient, req.body().nioByteBuffer(), new RpcResponseCallback() {
@Override
public void onSuccess(ByteBuffer response) {
respond(new RpcResponse(req.requestId, new NioManagedBuffer(response)));
}
@Override
public void onFailure(Throwable e) {
respond(new RpcFailure(req.requestId, Throwables.getStackTraceAsString(e)));
}
});
} catch (Exception e) {
logger.error("Error while invoking RpcHandler#receive() on RPC id " + req.requestId, e);
respond(new RpcFailure(req.requestId, Throwables.getStackTraceAsString(e)));
} finally {
req.body().release();
}
}
override def receive(
client: TransportClient,
message: ByteBuffer,
callback: RpcResponseCallback): Unit = {
val messageToDispatch = internalReceive(client, message)
// dispatcher负责发送远程的消息,最终调到postMessage 方法
dispatcher.postRemoteMessage(messageToDispatch, callback)
}
继续跟进 dispatcher.postRemoteMessage 方法:
/** Posts a message sent by a remote endpoint. */
def postRemoteMessage(message: RequestMessage, callback: RpcResponseCallback): Unit = {
val rpcCallContext =
new RemoteNettyRpcCallContext(nettyEnv, callback, message.senderAddress)
val rpcMessage = RpcMessage(message.senderAddress, message.content, rpcCallContext)
postMessage(message.receiver.name, rpcMessage, (e) => callback.onFailure(e))
}
在 postRemoteMessage 中,无论是请求消息还是回应消息,都最终会执行到这个 postMessage:
/**
* Posts a message to a specific endpoint.
*
* @param endpointName name of the endpoint.
* @param message the message to post
* @param callbackIfStopped callback function if the endpoint is stopped.
*/
private def postMessage(
endpointName: String,
message: InboxMessage,
callbackIfStopped: (Exception) => Unit): Unit = {
val error = synchronized {
// 获取消息的通信端邮箱名称
val data = endpoints.get(endpointName)
if (stopped) {
Some(new RpcEnvStoppedException())
} else if (data == null) {
Some(new SparkException(s"Could not find $endpointName."))
} else {
// 消息放入通信端的消息队列中
data.inbox.post(message)
//
receivers.offer(data)
None
}
}
// We don't need to call `onStop` in the `synchronized` block
error.foreach(callbackIfStopped)
}
在该方法中会将 message 放入 inbox 中,在处理消息的程序启动后,就能接收到消息,至此 RpcEnv环境启动完毕,紧接着 Master 要在RpcEnv 中注册:
val masterEndpoint = rpcEnv.setupEndpoint(ENDPOINT_NAME,
new Master(rpcEnv, rpcEnv.address, webUiPort, securityMgr, conf))
Master 注册了自己 EndPoint,可以接受处理消息,Master启动成功。
Worker 同理,Spark 集群启动成功。
Spark 提交任务向 Master 申请启动 Driver
用户提交程序
用户编写Spark应用程序后,使用spark-submit 脚本去提交,执行命令为:
./bin/spark-submit --class <main-class> --master <master-url> --deploy-mode <deploy-mode> --conf <key>=<value> ... # other options
<application-jar> [application-arguments]
实例如下:
./bin/spark-submit --class org.apache.spark.examples.SparkPi --master spark://192.168.1.20:7077 --deploy-mode cluster --supervise --executor-memory 2G --total-executor-cores 5 /path/to/examples.jar 1000
详情参考官网:http://spark.apache.org/docs/latest/submitting-applications.html
spark-submit 脚本
位置:spark-2.4.5\bin\spark-submit
if [ -z "${SPARK_HOME}" ]; then
source "$(dirname "$0")"/find-spark-home
fi
exec "${SPARK_HOME}"/bin/spark-class org.apache.spark.deploy.SparkSubmit "$@"
/bin/spark-class脚本
位置:spark-2.4.5\bin\spark-class
-z 判断SPARK_HOME变量的长度是否为0,等于0为真
if [ -z "${SPARK_HOME}" ]; then
加载当前目录的find...的变量
source "$(dirname "$0")"/find-spark-home
fi
加载环境变量
. "${SPARK_HOME}"/bin/load-spark-env.sh
# Find the java binary
-n 判断变量长度是否不为0,不为0为真
if [ -n "${JAVA_HOME}" ]; then
JAVAHOME存在就赋值RUNNER为这个
RUNNER="${JAVA_HOME}/bin/java"
else
监测java命令是否存在
if [ "$(command -v java)" ]; then
RUNNER="java"
else
不存在退出
echo "JAVA_HOME is not set" >&2
exit 1
fi
fi
# Find Spark jars.
判断${SPARK_HOME}/jars目录是否存在,存在为真
if [ -d "${SPARK_HOME}/jars" ]; then
SPARK_JARS_DIR="${SPARK_HOME}/jars"
else
SPARK_JARS_DIR="${SPARK_HOME}/assembly/target/scala-$SPARK_SCALA_VERSION/jars"
fi
判断下边俩,不都存在就报错退出
if [ ! -d "$SPARK_JARS_DIR" ] && [ -z "$SPARK_TESTING$SPARK_SQL_TESTING" ]; then
echo "Failed to find Spark jars directory ($SPARK_JARS_DIR)." 1>&2
echo "You need to build Spark with the target \"package\" before running this program." 1>&2
exit 1
else
存在就变量赋值
LAUNCH_CLASSPATH="$SPARK_JARS_DIR/*"
fi
# For tests
这变量长度大于1 unset目录权限
if [[ -n "$SPARK_TESTING" ]]; then
unset YARN_CONF_DIR
unset HADOOP_CONF_DIR
fi
build_command() {
执行命令获取
"$RUNNER" -Xmx128m -cp "$LAUNCH_CLASSPATH" org.apache.spark.launcher.Main "$@"
输出返回值
printf "%d\0" $?
}
创建数组
CMD=()
把build_commands输出结果,循环加到数组CMD中
while IFS= read -d '' -r ARG; do
CMD+=("$ARG")
done < <(build_command "$@")
数组长度
COUNT=${#CMD[@]}
数组长度-1
LAST=$((COUNT - 1))
数组的最后一个值,也就是上边$?的值
LAUNCHER_EXIT_CODE=${CMD[$LAST]}
如果返回值不是数字,退出
if ! [[ $LAUNCHER_EXIT_CODE =~ ^[0-9]+$ ]]; then
echo "${CMD[@]}" | head -n-1 1>&2
exit 1
fi
如果返回值不为0,退出,返回返回值
if [ $LAUNCHER_EXIT_CODE != 0 ]; then
exit $LAUNCHER_EXIT_CODE
fi
CMD还是原来那些参数,$@
CMD=("${CMD[@]:0:$LAST}")
执行这些
exec "${CMD[@]}"
执行步骤:
- 校验 $SPARK_HOME/conf,spark相关依赖目录SPARK_HOME/jars
- 将校验所得所有目录地址拼接为LAUNCH_CLASSPATH变量
- 将$JAVA_HOME/bin/java 定义为RUNNER变量
- 调用build_command()方法,创建执行命令
- 把build_command()方法创建的命令,循环加到数组CMD中,最后执行exec执行CMD命令
最终,执行的CMD命令如下 :
/opt/jdk1.8/bin/java -Dhdp.version=2.6.0.3-8 -cp /usr/hdp/current/spark2-historyserver/conf/:/usr/hdp/2.6.0.3-8/spark2/jars/*:/usr/hdp/current/hadoop-client/conf org.apache.spark.deploy.SparkSubmit --master spark://192.168.1.20:7077 \
--deploy-mode cluster --class org.apache.spark.examples.SparkPi --executor-memory 2G --total-executor-cores 5 ../examples/jars/sparkexamples_2.11-2.1.0.2.6.0.3-8.jar 1000
最终指定了程序的入口org.apache.spark.deploy.SparkSubmit
SparkSubmit伴生对象的main方法实例一个伴生类对象submit,执行submit.doSubmit(args)
def doSubmit(args: Array[String]): Unit
-
解析参数,实例化new SparkSubmitArguments(args),构造函数中调用loadEnvironmentArguments(),该方法的目的是加载环境变量,若参数中无action变量则赋值为submit, action = Option(action).getOrElse(SUBMIT)
-
action变量模式匹配
appArgs.action match { case SparkSubmitAction.SUBMIT => submit(appArgs, uninitLog) case SparkSubmitAction.KILL => kill(appArgs) case SparkSubmitAction.REQUEST_STATUS => requestStatus(appArgs) case SparkSubmitAction.PRINT_VERSION => printVersion()
-
submit中调用doRunMain,而doRunMain调用runMain
-
runMain中val (childArgs, childClasspath, sparkConf, childMainClass) = prepareSubmitEnvironment(args)用来初始化变量
(childArgs, childClasspath, sparkConf, childMainClass) = prepareSubmitEnvironment(args)
- 根据参数中master和delpoy-mode,设置对应的clusterManager和部署模式
- 再根据args中的其他参数,设置相关childArgs, childClasspath, sysProps, childMainClass,并返回结果
重点注意 childMainClass 类,是最后启动 Driver 的类
假设部署模式为standalone-cluster ,
// In legacy standalone cluster mode, use Client as a wrapper around the user class
childMainClass = STANDALONE_CLUSTER_SUBMIT_CLASS
private[deploy] val STANDALONE_CLUSTER_SUBMIT_CLASS = classOf[ClientApp].getName()
// 加载类
mainClass = Utils.classForName(childMainClass)
val app: SparkApplication = if (classOf[SparkApplication].isAssignableFrom(mainClass)) {
mainClass.newInstance().asInstanceOf[SparkApplication]
这里首先将 childMainClass 加载出来,赋给变量 mainClass,之后会将 mainClass 映射成 SparkApplication
app.start(childArgs.toArray, sparkConf)
接下来调用start方法,这里调用的是 ClientApp 的 start 方法,因为 childMainClass 是 clientApp 的类型:
private[spark] class ClientApp extends SparkApplication {
override def start(args: Array[String], conf: SparkConf): Unit = {
val driverArgs = new ClientArguments(args)
if (!conf.contains("spark.rpc.askTimeout")) {
conf.set("spark.rpc.askTimeout", "10s")
}
Logger.getRootLogger.setLevel(driverArgs.logLevel)
val rpcEnv =
RpcEnv.create("driverClient", Utils.localHostName(), 0, conf, new SecurityManager(conf))
val masterEndpoints = driverArgs.masters.map(RpcAddress.fromSparkURL).
map(rpcEnv.setupEndpointRef(_, Master.ENDPOINT_NAME))
rpcEnv.setupEndpoint("client", new ClientEndpoint(rpcEnv, driverArgs, masterEndpoints, conf))
rpcEnv.awaitTermination()
}
在 rpc 中设置提交当前任务的 Endpoint,只要设置肯定会运行 new ClientEndpoint 类的 onStart 方法:
val command = new Command(mainClass,
Seq("{{WORKER_URL}}", "{{USER_JAR}}", driverArgs.mainClass) ++ driverArgs.driverOptions,
sys.env, classPathEntries, libraryPathEntries, javaOpts)
val driverDescription = new DriverDescription(
driverArgs.jarUrl,
driverArgs.memory,
driverArgs.cores,
driverArgs.supervise,
command)
asyncSendToMasterAndForwardReply[SubmitDriverResponse](
RequestSubmitDriver(driverDescription))
在此方法中发送RequestSubmitDrive消息,对应于橙色2
为什么一定会调用onStart ?
/** Message loop used for dispatching messages. */
private class MessageLoop extends Runnable {
override def run(): Unit = {
try {
while (true) {
try {
/**
* 通过消息队列receivers取出一条消息,该消息的类型为EndpointData,内部都封装了inbox实例,
* 因此直接调用该实例的process函数去处理消息,
* 在集群启动的过程中,master向消息容器中放入一条onStart消息,因此取出的第一条消息为onStart
*/
val data = receivers.take()
if (data == PoisonPill) {
// Put PoisonPill back so that other MessageLoops can see it.
receivers.offer(PoisonPill)
return
}
data.inbox.process(Dispatcher.this)
} catch {
case NonFatal(e) => logError(e.getMessage, e)
}
}
} catch {
case _: InterruptedException => // exit
case t: Throwable =>
try {
// Re-submit a MessageLoop so that Dispatcher will still work if
// UncaughtExceptionHandler decides to not kill JVM.
threadpool.execute(new MessageLoop)
} finally {
throw t
}
}
}
// 调用Endpoint 的onStart函数
case OnStart =>
endpoint.onStart()
if (!endpoint.isInstanceOf[ThreadSafeRpcEndpoint]) {
inbox.synchronized {
if (!stopped) {
enableConcurrent = true
}
}
}
这里将 org.apache.spark.deploy.worker.DriverWrapper 封装成 command,并且将 command 封装到 driverDescription 中,然后向 Master 申请启动 Driver :
在master端的MessageLoop的run方法中取出消息,并调用data.inbox.process(Dispatcher.this),跟进process方法:
/**
* Process stored messages.
*/
def process(dispatcher: Dispatcher): Unit = {
var message: InboxMessage = null
inbox.synchronized {
if (!enableConcurrent && numActiveThreads != 0) {
return
}
message = messages.poll()
if (message != null) {
numActiveThreads += 1
} else {
return
}
}
while (true) {
safelyCall(endpoint) {
message match {
case RpcMessage(_sender, content, context) =>
try {
endpoint.receiveAndReply(context).applyOrElse[Any, Unit](content, { msg =>
throw new SparkException(s"Unsupported message $message from ${_sender}")
})
Master 中的 receiveAndReply 方法会接收此请求消息,跟进 receiveAndReply:
override def receiveAndReply(context: RpcCallContext): PartialFunction[Any, Unit] = {
case RequestSubmitDriver(description) =>
if (state != RecoveryState.ALIVE) {
val msg = s"${Utils.BACKUP_STANDALONE_MASTER_PREFIX}: $state. " +
"Can only accept driver submissions in ALIVE state."
context.reply(SubmitDriverResponse(self, false, None, msg))
} else {
logInfo("Driver submitted " + description.command.mainClass)
val driver = createDriver(description)
persistenceEngine.addDriver(driver)
waitingDrivers += driver
drivers.add(driver)
schedule()
// TODO: It might be good to instead have the submission client poll the master to determine
// the current status of the driver. For now it's simply "fire and forget".
context.reply(SubmitDriverResponse(self, true, Some(driver.id),
s"Driver successfully submitted as ${driver.id}"))
}
这里首先会判断 Master 的状态,如果符合要求的话,会使用之前封装的 description 对象创建 driver,driver 其实是一个 DriverInfo 的类型,里面封装了一些 Driver 的信息 ; 然后返回SubmitDriverResponse消息,对应橙色3
- 回到Client端,在inbox中可以看到:
case OneWayMessage(_sender, content) =>
endpoint.receive.applyOrElse[Any, Unit](content, { msg =>
throw new SparkException(s"Unsupported message $message from ${_sender}")
})
跟进Client端的receive方法:
override def receive: PartialFunction[Any, Unit] = {
case SubmitDriverResponse(master, success, driverId, message) =>
logInfo(message)
if (success) {
activeMasterEndpoint = master
pollAndReportStatus(driverId.get)
} else if (!Utils.responseFromBackup(message)) {
System.exit(-1)
}
若消息的类型为SubmitDriverResponse,则调用 pollAndReportStatus(driverId.get)
/* Find out driver status then exit the JVM */
def pollAndReportStatus(driverId: String): Unit = {
// Since ClientEndpoint is the only RpcEndpoint in the process, blocking the event loop thread
// is fine.
logInfo("... waiting before polling master for driver state")
Thread.sleep(5000)
logInfo("... polling master for driver state")
// 发送RequestDriverStatus, 对应橙色4
val statusResponse =
activeMasterEndpoint.askSync[DriverStatusResponse](RequestDriverStatus(driverId))
if (statusResponse.found) {
logInfo(s"State of $driverId is ${statusResponse.state.get}")
// Worker node, if present
(statusResponse.workerId, statusResponse.workerHostPort, statusResponse.state) match {
case (Some(id), Some(hostPort), Some(DriverState.RUNNING)) =>
logInfo(s"Driver running on $hostPort ($id)")
case _ =>
}
// Exception, if present
statusResponse.exception match {
case Some(e) =>
logError(s"Exception from cluster was: $e")
e.printStackTrace()
System.exit(-1)
case _ =>
System.exit(0)
}
} else {
logError(s"ERROR: Cluster master did not recognize $driverId")
System.exit(-1)
}
}
发送RequestDriverStatus, 对应橙色4;退出jvm进程,对应橙色5
- 在上述master接收到RequestSubmitDriver消息后还看到在waitingDrivers (private val waitingDrivers = new ArrayBuffer[DriverInfo]) 中添加刚才创建完的 DriverInfo对象,进入 schedule() 调度方法:
/**
* Schedule the currently available resources among waiting apps. This method will be called
* every time a new app joins or resource availability changes.
*/
private def schedule(): Unit = {
// 判断master的状态
if (state != RecoveryState.ALIVE) {
return
}
// Drivers take strict precedence over executors
val shuffledAliveWorkers = Random.shuffle(workers.toSeq.filter(_.state == WorkerState.ALIVE))
// 可用的worker数量
val numWorkersAlive = shuffledAliveWorkers.size
var curPos = 0
// 取出waitingDrivers中的driver信息
for (driver <- waitingDrivers.toList) { // iterate over a copy of waitingDrivers
// We assign workers to each waiting driver in a round-robin fashion. For each driver, we
// start from the last worker that was assigned a driver, and continue onwards until we have
// explored all alive workers.
var launched = false
var numWorkersVisited = 0
while (numWorkersVisited < numWorkersAlive && !launched) {
val worker = shuffledAliveWorkers(curPos)
numWorkersVisited += 1
/**
* worker中的cpu和内存资源与driver中描述的进行比较,若资源足够则调用launchDriver
*
*/
if (worker.memoryFree >= driver.desc.mem && worker.coresFree >= driver.desc.cores) {
launchDriver(worker, driver)
waitingDrivers -= driver
launched = true
}
curPos = (curPos + 1) % numWorkersAlive
}
}
/**
* 在worker上启动Executor进程
*/
startExecutorsOnWorkers()
}
launchDriver对应紫色1
private def launchDriver(worker: WorkerInfo, driver: DriverInfo) {
logInfo("Launching driver " + driver.id + " on worker " + worker.id)
worker.addDriver(driver)
driver.worker = Some(worker)
// 给worker上的endpoint发送LaunchDriver消息
worker.endpoint.send(LaunchDriver(driver.id, driver.desc))
driver.state = DriverState.RUNNING
}
spark-2.4.5\core\src\main\scala\org\apache\spark\deploy\worker\Worker.scala中的receive 方法匹配 LaunchDriver,实例化DriverRunner实例,对应紫色2
case LaunchDriver(driverId, driverDesc) =>
logInfo(s"Asked to launch driver $driverId")
val driver = new DriverRunner(
conf,
driverId,
workDir,
sparkHome,
driverDesc.copy(command = Worker.maybeUpdateSSLSettings(driverDesc.command, conf)),
self,
workerUri,
securityMgr)
drivers(driverId) = driver
// 启动Driver,会初始化 org.apache.spark.deploy.worker.DriverWrapper ,运行main方法
driver.start()
coresUsed += driverDesc.cores
memoryUsed += driverDesc.mem
这里说的启动的 Driver 就是刚才说的 val mainClass = “org.apache.spark.deploy.worker.DriverWrapper”,Driver 启动的就是 DriverWrapper 类的启动,DriverWrapper 的启动就是在 Worker 中创建一个 Driver 进程。启动Driver,会初始化org.apache.spark.deploy.worker.DriverWrapper,运行 main 方法,对应紫色3此方法中的mainClass,就是我们真正提交的 Application
// Delegate to supplied main class
val clazz = Utils.classForName(mainClass)
// 得到提交的application的主方法
val mainMethod = clazz.getMethod("main", classOf[Array[String]])
/**
* 启动用户提交的application中的main方法
* 由于application中需要实例化SparkContext对象
*/
mainMethod.invoke(null, extraArgs.toArray[String])
Spark Driver 启动,并向 Master 注册 Applciation
当在 Worker 启动完 Driver 之后 ,实例化SparkContext对象,在源码SparkContext 类中,可以找到:
- 根据sparkConf创建SparkEnv,对应红色1
// Create the Spark execution environment (cache, map output tracker, etc)
_env = createSparkEnv(_conf, isLocal, listenerBus)
- 创建TaskScheduler和StandaloneSchedulerBackend ,对应红色2和红色3
/**
* sched, ts分别对应StandaloneSchedulerBackend 和 TaskSchedulerImpl 两个对象
*/
// Create and start the scheduler
val (sched, ts) = SparkContext.createTaskScheduler(this, master, deployMode)
_schedulerBackend = sched
_taskScheduler = ts
_dagScheduler = new DAGScheduler(this)
_heartbeatReceiver.ask[Boolean](TaskSchedulerIsSet)
// start TaskScheduler after taskScheduler sets DAGScheduler reference in DAGScheduler's
// constructor
// TaskSchedulerImpl 对象的start方法
_taskScheduler.start()
跟进createTaskScheduler:
// standalone 提交任务都是以 “spark://”这种方式提交
case SPARK_REGEX(sparkUrl) =>
//scheduler 创建TaskSchedulerImpl 对象
val scheduler = new TaskSchedulerImpl(sc)
val masterUrls = sparkUrl.split(",").map("spark://" + _)
val backend = new StandaloneSchedulerBackend(scheduler, sc, masterUrls)
//调用TaskSchedulerImpl对象中的initialize方,使用backend将其初始化
scheduler.initialize(backend)
(backend, scheduler)
因为是以Standalone的方式提交任务,所以找到匹配的 case。在这里创建了 TaskSchedulerImpl 的对象scheduler,并将其传入了 StandaloneSchedulerBackend(scheduler, sc, masterUrls)中,得知,返回的 backend 是一个 Standalone 大环境的任务调度器,scheduler 则是 TaskScheduler 的调度器。
def initialize(backend: SchedulerBackend) {
this.backend = backend
schedulableBuilder = {
schedulingMode match {
case SchedulingMode.FIFO =>
new FIFOSchedulableBuilder(rootPool)
case SchedulingMode.FAIR =>
new FairSchedulableBuilder(rootPool, conf)
case _ =>
throw new IllegalArgumentException(s"Unsupported $SCHEDULER_MODE_PROPERTY: " +
s"$schedulingMode")
}
}
schedulableBuilder.buildPools()
}
在 scheduler.initialize(backend) 中的作用就是将 backend 设置为 scheduler 的属性,且StandaloneSchedulerBackend 继承了CoarseGrainedSchedulerBackend 类。
- 回到TaskSchedulerImpl 对象的start方法
override def start() {
backend.start()
if (!isLocal && conf.getBoolean("spark.speculation", false)) {
logInfo("Starting speculative execution thread")
speculationScheduler.scheduleWithFixedDelay(new Runnable {
override def run(): Unit = Utils.tryOrStopSparkContext(sc) {
checkSpeculatableTasks()
}
}, SPECULATION_INTERVAL_MS, SPECULATION_INTERVAL_MS, TimeUnit.MILLISECONDS)
}
}
start方法就是调用backend.start() 方法 ,因为 backend 是 StandaloneSchedulerBackend 类型,所有调用的是 StandaloneSchedulerBackend 中的 start() 方法 :
override def start() {
super.start()
// SPARK-21159. The scheduler backend should only try to connect to the launcher when in client
// mode. In cluster mode, the code that submits the application to the Master needs to connect
// to the launcher instead.
if (sc.deployMode == "client") {
launcherBackend.connect()
}
进入 super.start(),在方法中向 RpcEnv 注册了 Driver 端的 Endpoint(先创建,对应红色4):
override def start() {
val properties = new ArrayBuffer[(String, String)]
for ((key, value) <- scheduler.sc.conf.getAll) {
if (key.startsWith("spark.")) {
properties += ((key, value))
}
}
// TODO (prashant) send conf instead of properties
/**
* 创建Driver的Endpoint
*/
driverEndpoint = createDriverEndpointRef(properties)
}
// 向rpcEnv中注册当前DriverEndpoint
protected def createDriverEndpointRef(
properties: ArrayBuffer[(String, String)]): RpcEndpointRef = {
rpcEnv.setupEndpoint(ENDPOINT_NAME, createDriverEndpoint(properties))
}
protected def createDriverEndpoint(properties: Seq[(String, String)]): DriverEndpoint = {
new DriverEndpoint(rpcEnv, properties)
}
backend.start() 方法执行创建完 DriverEndpoint 之后,还执行了如下代码,功能是向 Driver 注册application 的信息 :
val appDesc = ApplicationDescription(sc.appName, maxCores, sc.executorMemory, command,
webUrl, sc.eventLogDir, sc.eventLogCodec, coresPerExecutor, initialExecutorLimit)
client = new StandaloneAppClient(sc.env.rpcEnv, masters, appDesc, this, conf)
client.start()
launcherBackend.setState(SparkAppHandle.State.SUBMITTED)
waitForRegistration()
launcherBackend.setState(SparkAppHandle.State.RUNNING)
创建 StandaloneAppClient对象,对应红色5,进入client.start() 后就是在rpcEnv 上注册信息 :
def start() {
// Just launch an rpcEndpoint; it will call back into the listener.
/**
* rpcEnv.setupEndpoint中创建了ClientEndpoint
* 只要设置Endpoint肯定会调用 ClientEndpoint的onStart方法
*/
endpoint.set(rpcEnv.setupEndpoint("AppClient", new ClientEndpoint(rpcEnv)))
}
创建ClientEndpoint对应红色6
跟进ClientEndpoint的onStart() 方法
override def onStart(): Unit = {
try {
// 向master注册application信息
registerWithMaster(1)
} catch {
case e: Exception =>
logWarning("Failed to connect to master", e)
markDisconnected()
stop()
}
}
注册信息对应红色7
向 Master 注册当前application的信息的 registerWithMaster() 方法,因为我们的 Master 会有高可用,所以要给所有的 Master 注册,进入后看到 tryRegisterAllMasters():
/**
* Register with all masters asynchronously. It will call `registerWithMaster` every
* REGISTRATION_TIMEOUT_SECONDS seconds until exceeding REGISTRATION_RETRIES times.
* Once we connect to a master successfully, all scheduling work and Futures will be cancelled.
*
* nthRetry means this is the nth attempt to register with master.
*/
private def registerWithMaster(nthRetry: Int) {
// tryRegisterAllMasters 尝试向所有的Master 注册application信息
registerMasterFutures.set(tryRegisterAllMasters())
registrationRetryTimer.set(registrationRetryThread.schedule(new Runnable {
override def run(): Unit = {
if (registered.get) {
registerMasterFutures.get.foreach(_.cancel(true))
registerMasterThreadPool.shutdownNow()
} else if (nthRetry >= REGISTRATION_RETRIES) {
markDead("All masters are unresponsive! Giving up.")
} else {
registerMasterFutures.get.foreach(_.cancel(true))
registerWithMaster(nthRetry + 1)
}
}
}, REGISTRATION_TIMEOUT_SECONDS, TimeUnit.SECONDS))
}
进入 tryRegisterAllMasters() 之后,会获取 Master 的 Endpoint,向 Master 注册 application,Master 类中receive 方法中会匹配接收 RegisterApplication 类型,随后在 Master 中匹配 RegisterApplication 的 case:
/**
* Register with all masters asynchronously and returns an array `Future`s for cancellation.
*/
private def tryRegisterAllMasters(): Array[JFuture[_]] = {
for (masterAddress <- masterRpcAddresses) yield {
registerMasterThreadPool.submit(new Runnable {
override def run(): Unit = try {
if (registered.get) {
return
}
logInfo("Connecting to master " + masterAddress.toSparkURL + "...")
// 获取到master上Endpoint的引用
val masterRef = rpcEnv.setupEndpointRef(masterAddress, Master.ENDPOINT_NAME)
// 向master发送RegisterApplication信息,Master类中receive方法中会匹配接收RegisterApplication类型
masterRef.send(RegisterApplication(appDescription, self))
} catch {
case ie: InterruptedException => // Cancelled
case NonFatal(e) => logWarning(s"Failed to connect to master $masterAddress", e)
}
})
}
}
Master 匹配 RegisterApplication 类型消息的处理流程:
case RegisterApplication(description, driver) =>
// TODO Prevent repeated registrations from some driver
// 如果master的状态是standby 忽略
if (state == RecoveryState.STANDBY) {
// ignore, don't send response
} else {
logInfo("Registering app " + description.name)
// 封装application信息
val app = createApplication(description, driver)
// 注册app
registerApplication(app)
logInfo("Registered app " + description.name + " with ID " + app.id)
persistenceEngine.addApplication(app)
// 向Driver发送RegisteredApplication消息
driver.send(RegisteredApplication(app.id, self))
// 又执行通用方法schedule()
schedule()
}
此处对应红色8
至此 Driver 向 Master 注册 Application 流程结束。
Master 发送消息启动 Executor
紧接上述的 schedule()方法,之前是从 launchDriver(worker, driver) 进去的,现在又出来继续调用 startExecutorsOnWorkers() 方法:
/**
* Schedule and launch executors on workers
*/
private def startExecutorsOnWorkers(): Unit = {
// Right now this is a very simple FIFO scheduler. We keep trying to fit in the first app
// in the queue, then the second app, etc.
// 从waitingApps中获取提交的app
for (app <- waitingApps) {
// 在application中获取启动一个Executor使用的core数量
val coresPerExecutor = app.desc.coresPerExecutor.getOrElse(1)
// If the cores left is less than the coresPerExecutor,the cores left will not be allocated
if (app.coresLeft >= coresPerExecutor) {
// Filter out workers that don't have enough resources to launch an executor
val usableWorkers = workers.toArray.filter(_.state == WorkerState.ALIVE)
.filter(worker => worker.memoryFree >= app.desc.memoryPerExecutorMB &&
worker.coresFree >= coresPerExecutor)
.sortBy(_.coresFree).reverse
/**
* 去worker中划分每个worker提供多少core和启动多少Executor,
* 返回的assignedCores是每个worker节点中应该给当前的application分配多少core
*/
val assignedCores = scheduleExecutorsOnWorkers(app, usableWorkers, spreadOutApps)
// Now that we've decided how many cores to allocate on each worker, let's allocate them
for (pos <- 0 until usableWorkers.length if assignedCores(pos) > 0) {
// 在worker中给Executor划分资源
allocateWorkerResourceToExecutors(
app, assignedCores(pos), app.desc.coresPerExecutor, usableWorkers(pos))
}
}
}
}
scheduleExecutorsOnWorkers(app, usableWorkers, spreadOutApps) 方法,最后返回最后返回每个Worker上分配多少core ;然后运行
allocateWorkerResourceToExecutors,传入当前 application,每个 worker 分配的核数,每个核上要启动的 Executor 数量以及可用的 usableWorkers
/**
* Allocate a worker's resources to one or more executors.
* @param app the info of the application which the executors belong to
* @param assignedCores number of cores on this worker for this application
* @param coresPerExecutor number of cores per executor
* @param worker the worker info
*/
private def allocateWorkerResourceToExecutors(
app: ApplicationInfo,
assignedCores: Int,
coresPerExecutor: Option[Int],
worker: WorkerInfo): Unit = {
// If the number of cores per executor is specified, we divide the cores assigned
// to this worker evenly among the executors with no remainder.
// Otherwise, we launch a single executor that grabs all the assignedCores on this worker.
val numExecutors = coresPerExecutor.map { assignedCores / _ }.getOrElse(1)
// 每个Executor要分配多少个core
val coresToAssign = coresPerExecutor.getOrElse(assignedCores)
for (i <- 1 to numExecutors) {
val exec = app.addExecutor(worker, coresToAssign)
// worker中启动Executor
launchExecutor(worker, exec)
app.state = ApplicationState.RUNNING
}
}
在 launchExecutor(worker, exec) 方法中,会获取 Worker 的通信邮箱,给 Worker 发送启动 Executor 的消息,具体就是启动Executor需要多少个 core 和 内存,然后在 Worker 中有 receive 方法一直匹配 LaunchExecutor 类型,对应蓝色1:
private def launchExecutor(worker: WorkerInfo, exec: ExecutorDesc): Unit = {
logInfo("Launching executor " + exec.fullId + " on worker " + worker.id)
worker.addExecutor(exec)
/**
* 获取worker的通信邮箱,向其发送LaunchExecutor消息
*/
worker.endpoint.send(LaunchExecutor(masterUrl,
exec.application.id, exec.id, exec.application.desc, exec.cores, exec.memory))
/**
* 获取driver的通信邮箱,向其发送ExecutorAdded消息
*/
exec.application.driver.send(
ExecutorAdded(exec.id, worker.id, worker.hostPort, exec.cores, exec.memory))
}
worker接收到消息,进行蓝色2和蓝色3的操作:
// 接收到LaunchExecutor消息
case LaunchExecutor(masterUrl, appId, execId, appDesc, cores_, memory_) =>
if (masterUrl != activeMasterUrl) {
logWarning("Invalid Master (" + masterUrl + ") attempted to launch executor.")
} else {
try {
logInfo("Asked to launch executor %s/%d for %s".format(appId, execId, appDesc.name))
// Create the executor's working directory
val executorDir = new File(workDir, appId + "/" + execId)
if (!executorDir.mkdirs()) {
throw new IOException("Failed to create directory " + executorDir)
}
// Create local dirs for the executor. These are passed to the executor via the
// SPARK_EXECUTOR_DIRS environment variable, and deleted by the Worker when the
// application finishes.
val appLocalDirs = appDirectories.getOrElse(appId, {
val localRootDirs = Utils.getOrCreateLocalRootDirs(conf)
val dirs = localRootDirs.flatMap { dir =>
try {
val appDir = Utils.createDirectory(dir, namePrefix = "executor")
Utils.chmod700(appDir)
Some(appDir.getAbsolutePath())
} catch {
case e: IOException =>
logWarning(s"${e.getMessage}. Ignoring this directory.")
None
}
}.toSeq
if (dirs.isEmpty) {
throw new IOException("No subfolder can be created in " +
s"${localRootDirs.mkString(",")}.")
}
dirs
})
appDirectories(appId) = appLocalDirs
// 创建ExecutorRunner对象
val manager = new ExecutorRunner(
appId,
execId,
/**
* appDesc中有 Command("org.apache.spark.executor.CoarseGrainedExecutorBackend",.......)
* 第一个参数就是Executor类
*/
appDesc.copy(command = Worker.maybeUpdateSSLSettings(appDesc.command, conf)),
cores_,
memory_,
self,
workerId,
host,
webUi.boundPort,
publicAddress,
sparkHome,
executorDir,
workerUri,
conf,
appLocalDirs, ExecutorState.RUNNING)
executors(appId + "/" + execId) = manager
// 启动ExecutorRunner对象
manager.start()
coresUsed += cores_
memoryUsed += memory_
// 向master发送ExecutorStateChanged消息
sendToMaster(ExecutorStateChanged(appId, execId, manager.state, None, None))
通过 manager 启动Executor,启动的就是 CoarseGrainedExecutorBackend 类,下面看 CoarseGrainedExecutorBackend 类中的 main 方法有反向注册给Driver:
val env = SparkEnv.createExecutorEnv(
driverConf, executorId, hostname, cores, cfg.ioEncryptionKey, isLocal = false)
//注册Executor的通信邮箱,会调用CoarseGrainedExecutorBackend的onstart方法
env.rpcEnv.setupEndpoint("Executor", new CoarseGrainedExecutorBackend(
env.rpcEnv, driverUrl, executorId, hostname, cores, userClassPath, env))
workerUrl.foreach { url =>
env.rpcEnv.setupEndpoint("WorkerWatcher", new WorkerWatcher(env.rpcEnv, url))
}
env.rpcEnv.awaitTermination()
会调用 CoarseGrainedExecutorBackend 类的 onStart() 方法:
override def onStart() {
logInfo("Connecting to driver: " + driverUrl)
//从RPC中拿到Driver的引用,给Driver反向注册Executor
rpcEnv.asyncSetupEndpointRefByURI(driverUrl).flatMap { ref =>
// This is a very fast action so we can use "ThreadUtils.sameThread"
//拿到Driver的引用
driver = Some(ref)
/**
* 给Driver反向注册Executor信息,这里就是注册给之前看到的 CoarseGrainedSchedulerBackend 类中的DriverEndpoint
* DriverEndpoint类中会有receiveAndReply 方法来匹配RegisterExecutor
*/
ref.ask[Boolean](RegisterExecutor(executorId, self, hostname, cores, extractLogUrls))
}(ThreadUtils.sameThread).onComplete {
// This is a very fast action so we can use "ThreadUtils.sameThread"
case Success(msg) =>
// Always receive `true`. Just ignore it
case Failure(e) =>
exitExecutor(1, s"Cannot register with driver: $driverUrl", e, notifyDriver = false)
}(ThreadUtils.sameThread)
}
在该方法中从 RPC 中拿到 Driver 的引用,将 Executor 反向注册给 Driver,方法中的 ref 就是 CoarseGrainedSchedulerBackend 类的引用,之后在 CoarseGrainedSchedulerBackend 中找到匹配 RegisterExecutor 的 case,用于反向注册:
/**
* 拿到Execuotr的通信邮箱,发送消息给ExecutorRef 告诉 Executor已经被注册。
* 在 CoarseGrainedExecutorBackend 类中 receive方法一直监听有没有被注册,匹配上就会启动Executor
*/
executorRef.send(RegisteredExecutor)
在 Driver 端告诉 Execuotr 端已经被注册,匹配上就会启动 Executor。去看 Executor 端匹配 RegisteredExecutor 的 case,用于启动 Executor:
override def receive: PartialFunction[Any, Unit] = {
//匹配上Driver端发过来的消息,已经接受注册Executor了,启动Executor
case RegisteredExecutor =>
logInfo("Successfully registered with driver")
try {
executor = new Executor(executorId, hostname, env, userClassPath, isLocal = false)
} catch {
case NonFatal(e) =>
exitExecutor(1, "Unable to create executor due to " + e.getMessage, e)
}
Executor 在被告知反向注册成功之后,开始真正的创建 Executor,Executor 中有线程池用于task运行:
private val threadPool = {
val threadFactory = new ThreadFactoryBuilder()
.setDaemon(true)
.setNameFormat("Executor task launch worker-%d")
.setThreadFactory(new ThreadFactory {
override def newThread(r: Runnable): Thread =
// Use UninterruptibleThread to run tasks so that we can allow running codes without being
// interrupted by `Thread.interrupt()`. Some issues, such as KAFKA-1894, HADOOP-10622,
// will hang forever if some methods are interrupted.
new UninterruptibleThread(r, "unused") // thread name will be set by ThreadFactoryBuilder
})
.build()
Executors.newCachedThreadPool(threadFactory).asInstanceOf[ThreadPoolExecutor]
}
至此,Executor 创建完毕。
master接收到ExecutorStateChanged,进行蓝色4的处理:
// master接收到ExecutorStateChanged消息
case ExecutorStateChanged(appId, execId, state, message, exitStatus) =>
val execOption = idToApp.get(appId).flatMap(app => app.executors.get(execId))
execOption match {
case Some(exec) =>
val appInfo = idToApp(appId)
val oldState = exec.state
exec.state = state
if (state == ExecutorState.RUNNING) {
assert(oldState == ExecutorState.LAUNCHING,
s"executor $execId state transfer from $oldState to RUNNING is illegal")
appInfo.resetRetryCount()
}
//向driver发送ExecutorUpdated消息
exec.application.driver.send(ExecutorUpdated(execId, state, message, exitStatus, false))
spark 任务调度
Executor 创建完毕 ,开始执行调度 ,用户提交的程序需要有一个action算子来触发计算,不妨以count()为例:
位置:spark-2.4.5\core\src\main\scala\org\apache\spark\rdd\RDD.scala
/**
* Return the number of elements in the RDD.
*/
def count(): Long = sc.runJob(this, Utils.getIteratorSize _).sum
sc 是 SparkContext 的对象,进入 runJob 方法:
/**
* Run a job on all partitions in an RDD and return the results in an array.
*
* @param rdd target RDD to run tasks on
* @param func a function to run on each partition of the RDD
* @return in-memory collection with a result of the job (each collection element will contain
* a result from one partition)
*/
def runJob[T, U: ClassTag](rdd: RDD[T], func: Iterator[T] => U): Array[U] = {
runJob(rdd, func, 0 until rdd.partitions.length)
}
再次进入 runJob 方法:
/**
* Run a function on a given set of partitions in an RDD and return the results as an array.
*
* @param rdd target RDD to run tasks on
* @param func a function to run on each partition of the RDD
* @param partitions set of partitions to run on; some jobs may not want to compute on all
* partitions of the target RDD, e.g. for operations like `first()`
* @return in-memory collection with a result of the job (each collection element will contain
* a result from one partition)
*/
def runJob[T, U: ClassTag](
rdd: RDD[T],
func: Iterator[T] => U,
partitions: Seq[Int]): Array[U] = {
val cleanedFunc = clean(func)
runJob(rdd, (ctx: TaskContext, it: Iterator[T]) => cleanedFunc(it), partitions)
}
再次进入 runJob 方法:
/**
* Run a function on a given set of partitions in an RDD and return the results as an array.
* The function that is run against each partition additionally takes `TaskContext` argument.
*
* @param rdd target RDD to run tasks on
* @param func a function to run on each partition of the RDD
* @param partitions set of partitions to run on; some jobs may not want to compute on all
* partitions of the target RDD, e.g. for operations like `first()`
* @return in-memory collection with a result of the job (each collection element will contain
* a result from one partition)
*/
def runJob[T, U: ClassTag](
rdd: RDD[T],
func: (TaskContext, Iterator[T]) => U,
partitions: Seq[Int]): Array[U] = {
val results = new Array[U](partitions.size)
runJob[T, U](rdd, func, partitions, (index, res) => results(index) = res)
results
}
再进此进入 runJob 方法,发现了 rdd.doCheckpoint() 方法,该方法的功能就是对整个 RDD 进行回溯;在 runJob 中传了一个参数 rdd,跟进该参数,在 dagScheduler.runJob(rdd, cleanedFunc, partitions, callSite, resultHandler, localProperties.get) 中调用:
/**
* Run a function on a given set of partitions in an RDD and pass the results to the given
* handler function. This is the main entry point for all actions in Spark.
*
* @param rdd target RDD to run tasks on
* @param func a function to run on each partition of the RDD
* @param partitions set of partitions to run on; some jobs may not want to compute on all
* partitions of the target RDD, e.g. for operations like `first()`
* @param resultHandler callback to pass each result to
*/
def runJob[T, U: ClassTag](
rdd: RDD[T],
func: (TaskContext, Iterator[T]) => U,
partitions: Seq[Int],
resultHandler: (Int, U) => Unit): Unit = {
if (stopped.get()) {
throw new IllegalStateException("SparkContext has been shutdown")
}
val callSite = getCallSite
val cleanedFunc = clean(func)
logInfo("Starting job: " + callSite.shortForm)
if (conf.getBoolean("spark.logLineage", false)) {
logInfo("RDD's recursive dependencies:\n" + rdd.toDebugString)
}
dagScheduler.runJob(rdd, cleanedFunc, partitions, callSite, resultHandler, localProperties.get)
progressBar.foreach(_.finishAll())
rdd.doCheckpoint()
}
进入方法后继续跟进 rdd 这个参数,在 submitJob 中调用,跟进 submitJob 方法中的 rdd 参数在何时使用:
/**
* Run an action job on the given RDD and pass all the results to the resultHandler function as
* they arrive.
*
* @param rdd target RDD to run tasks on
* @param func a function to run on each partition of the RDD
* @param partitions set of partitions to run on; some jobs may not want to compute on all
* partitions of the target RDD, e.g. for operations like first()
* @param callSite where in the user program this job was called
* @param resultHandler callback to pass each result to
* @param properties scheduler properties to attach to this job, e.g. fair scheduler pool name
*
* @note Throws `Exception` when the job fails
*/
def runJob[T, U](
rdd: RDD[T],
func: (TaskContext, Iterator[T]) => U,
partitions: Seq[Int],
callSite: CallSite,
resultHandler: (Int, U) => Unit,
properties: Properties): Unit = {
val start = System.nanoTime
val waiter = submitJob(rdd, func, partitions, callSite, resultHandler, properties)
ThreadUtils.awaitReady(waiter.completionFuture, Duration.Inf)
waiter.completionFuture.value.get match {
case scala.util.Success(_) =>
logInfo("Job %d finished: %s, took %f s".format
(waiter.jobId, callSite.shortForm, (System.nanoTime - start) / 1e9))
case scala.util.Failure(exception) =>
logInfo("Job %d failed: %s, took %f s".format
(waiter.jobId, callSite.shortForm, (System.nanoTime - start) / 1e9))
// SPARK-8644: Include user stack trace in exceptions coming from DAGScheduler.
val callerStackTrace = Thread.currentThread().getStackTrace.tail
exception.setStackTrace(exception.getStackTrace ++ callerStackTrace)
throw exception
}
}
/**
* Submit an action job to the scheduler.
*
* @param rdd target RDD to run tasks on
* @param func a function to run on each partition of the RDD
* @param partitions set of partitions to run on; some jobs may not want to compute on all
* partitions of the target RDD, e.g. for operations like first()
* @param callSite where in the user program this job was called
* @param resultHandler callback to pass each result to
* @param properties scheduler properties to attach to this job, e.g. fair scheduler pool name
*
* @return a JobWaiter object that can be used to block until the job finishes executing
* or can be used to cancel the job.
*
* @throws IllegalArgumentException when partitions ids are illegal
*/
def submitJob[T, U](
rdd: RDD[T],
func: (TaskContext, Iterator[T]) => U,
partitions: Seq[Int],
callSite: CallSite,
resultHandler: (Int, U) => Unit,
properties: Properties): JobWaiter[U] = {
// Check to make sure we are not launching a task on a partition that does not exist.
val maxPartitions = rdd.partitions.length
partitions.find(p => p >= maxPartitions || p < 0).foreach { p =>
throw new IllegalArgumentException(
"Attempting to access a non-existent partition: " + p + ". " +
"Total number of partitions: " + maxPartitions)
}
val jobId = nextJobId.getAndIncrement()
if (partitions.size == 0) {
// Return immediately if the job is running 0 tasks
return new JobWaiter[U](this, jobId, 0, resultHandler)
}
assert(partitions.size > 0)
val func2 = func.asInstanceOf[(TaskContext, Iterator[_]) => _]
val waiter = new JobWaiter(this, jobId, partitions.size, resultHandler)
//提交任务,eventProcessLoop是DAGSchedulerEventProcessLoop对象
eventProcessLoop.post(JobSubmitted(
jobId, rdd, func2, partitions.toArray, callSite, waiter,
SerializationUtils.clone(properties)))
waiter
}
rdd 传进了 eventProcessLoop.post 方法,进入 post 方法,该方法内部调用了 eventQueue.put(event),目的是将提交的任务放入队列:
/**
* Put the event into the event queue. The event thread will process it later.
*/
def post(event: E): Unit = {
//将提交的任务放入队列
eventQueue.put(event)
}
观察 eventQueue 类型:
private val eventQueue: BlockingQueue[E] = new LinkedBlockingDeque[E]()
eventQueue 存在于 EventLoop 类中,该类中有一个后台线程,从eventQueue中取出数据进行计算,运算逻辑在onReceive(event)
private[spark] val eventThread = new Thread(name) {
setDaemon(true)
override def run(): Unit = {
try {
while (!stopped.get) {
val event = eventQueue.take()
try {
onReceive(event)
} catch {
case NonFatal(e) =>
try {
onError(e)
} catch {
case NonFatal(e) => logError("Unexpected error in " + name, e)
}
}
}
进入该方法无法看到代码逻辑,去找 eventProcessLoop 的子类, DAGSchedulerEventProcessLoop :
private[scheduler] class DAGSchedulerEventProcessLoop(dagScheduler: DAGScheduler)
extends EventLoop[DAGSchedulerEvent]("dag-scheduler-event-loop") with Logging {
private[this] val timer = dagScheduler.metricsSource.messageProcessingTimer
/**
* The main event loop of the DAG scheduler.
*/
override def onReceive(event: DAGSchedulerEvent): Unit = {
val timerContext = timer.time()
try {
doOnReceive(event)
} finally {
timerContext.stop()
}
}
在 doOnReceive(event) 中运行 handleJobSubmitted 方法:
private def doOnReceive(event: DAGSchedulerEvent): Unit = event match {
case JobSubmitted(jobId, rdd, func, partitions, callSite, listener, properties) =>
dagScheduler.handleJobSubmitted(jobId, rdd, func, partitions, callSite, listener, properties)
在 handleJobSubmitted 方法中通过 submitStage(finalStage) 根据宽窄依赖递归寻找 Stage 并提交:
private[scheduler] def handleJobSubmitted(jobId: Int,
finalRDD: RDD[_],
func: (TaskContext, Iterator[_]) => _,
partitions: Array[Int],
callSite: CallSite,
listener: JobListener,
properties: Properties) {
var finalStage: ResultStage = null
try {
// New stage creation may throw an exception if, for example, jobs are run on a
// HadoopRDD whose underlying HDFS files have been deleted.
finalStage = createResultStage(finalRDD, func, partitions, jobId, callSite)
} catch {
case e: BarrierJobSlotsNumberCheckFailed =>
logWarning(s"The job $jobId requires to run a barrier stage that requires more slots " +
"than the total number of slots in the cluster currently.")
// If jobId doesn't exist in the map, Scala coverts its value null to 0: Int automatically.
val numCheckFailures = barrierJobIdToNumTasksCheckFailures.compute(jobId,
new BiFunction[Int, Int, Int] {
override def apply(key: Int, value: Int): Int = value + 1
})
if (numCheckFailures <= maxFailureNumTasksCheck) {
messageScheduler.schedule(
new Runnable {
override def run(): Unit = eventProcessLoop.post(JobSubmitted(jobId, finalRDD, func,
partitions, callSite, listener, properties))
},
timeIntervalNumTasksCheck,
TimeUnit.SECONDS
)
return
} else {
// Job failed, clear internal data.
barrierJobIdToNumTasksCheckFailures.remove(jobId)
listener.jobFailed(e)
return
}
case e: Exception =>
logWarning("Creating new stage failed due to exception - job: " + jobId, e)
listener.jobFailed(e)
return
}
// Job submitted, clear internal data.
barrierJobIdToNumTasksCheckFailures.remove(jobId)
val job = new ActiveJob(jobId, finalStage, callSite, listener, properties)
clearCacheLocs()
logInfo("Got job %s (%s) with %d output partitions".format(
job.jobId, callSite.shortForm, partitions.length))
logInfo("Final stage: " + finalStage + " (" + finalStage.name + ")")
logInfo("Parents of final stage: " + finalStage.parents)
logInfo("Missing parents: " + getMissingParentStages(finalStage))
val jobSubmissionTime = clock.getTimeMillis()
jobIdToActiveJob(jobId) = job
activeJobs += job
finalStage.setActiveJob(job)
val stageIds = jobIdToStageIds(jobId).toArray
val stageInfos = stageIds.flatMap(id => stageIdToStage.get(id).map(_.latestInfo))
listenerBus.post(
SparkListenerJobStart(job.jobId, jobSubmissionTime, stageInfos, properties))
// 递归寻找Stage
submitStage(finalStage)
}
/** Submits stage, but first recursively submits any missing parents. */
private def submitStage(stage: Stage) {
val jobId = activeJobForStage(stage)
if (jobId.isDefined) {
logDebug(s"submitStage($stage (name=${stage.name};" +
s"jobs=${stage.jobIds.toSeq.sorted.mkString(",")}))")
if (!waitingStages(stage) && !runningStages(stage) && !failedStages(stage)) {
val missing = getMissingParentStages(stage).sortBy(_.id)
logDebug("missing: " + missing)
if (missing.isEmpty) {
logInfo("Submitting " + stage + " (" + stage.rdd + "), which has no missing parents")
// 将stage划分为taskSet,并提交
submitMissingTasks(stage, jobId.get)
} else {
for (parent <- missing) {
// 递归实现
submitStage(parent)
}
waitingStages += stage
}
}
} else {
abortStage(stage, "No active job for stage " + stage.id, None)
}
}
划分完 Stage 之后,submitMissingTasks(stage, jobId.get) 方法会将 stage 划分成 task 发送到 Exeuctor 中执行,并以以 TaskSet 形式提交任务:
//以taskSet形式提交任务
taskScheduler.submitTasks(new TaskSet(
tasks.toArray, stage.id, stage.latestInfo.attemptNumber, jobId, properties))
提交task, TaskSchedulerImpl的submitTasks执行 backend.reviveOffers() 调用的是 CoarseGrainedSchedulerBackend 对象中的方法:
backend.reviveOffers()
override def reviveOffers() {
driverEndpoint.send(ReviveOffers)
}
对应粉色2
注:粉色1的流程如下
-
SparkContex的初始化过程中会调用TaskSchedulerImpl 对象的start方法,后者调用backend.start(),实质是调用CoarseGrainedSchedulerBackend.start(),
/** * 创建Driver的Endpoint */ driverEndpoint = createDriverEndpointRef(properties)
CoarseGrainedSchedulerBackend 中内部类DriverEndpoint.receive方法来接收数据 :
case ReviveOffers =>
makeOffers()
// Make fake resource offers on all executors
private def makeOffers() {
// Make sure no executor is killed while some task is launching on it
val taskDescs = withLock {
// Filter out executors under killing
val activeExecutors = executorDataMap.filterKeys(executorIsAlive)
val workOffers = activeExecutors.map {
case (id, executorData) =>
new WorkerOffer(id, executorData.executorHost, executorData.freeCores,
Some(executorData.executorAddress.hostPort))
}.toIndexedSeq
// TaskSchedulerImpl.resourceOffers方法
scheduler.resourceOffers(workOffers)
}
if (!taskDescs.isEmpty) {
// 启动任务
launchTasks(taskDescs)
}
}
对应粉色3
跟进launchTasks方法,对应粉色4
// Launch tasks returned by a set of resource offers
private def launchTasks(tasks: Seq[Seq[TaskDescription]]) {
for (task <- tasks.flatten) {
val serializedTask = TaskDescription.encode(task)
if (serializedTask.limit() >= maxRpcMessageSize) {
Option(scheduler.taskIdToTaskSetManager.get(task.taskId)).foreach { taskSetMgr =>
try {
var msg = "Serialized task %s:%d was %d bytes, which exceeds max allowed: " +
"spark.rpc.message.maxSize (%d bytes). Consider increasing " +
"spark.rpc.message.maxSize or using broadcast variables for large values."
msg = msg.format(task.taskId, task.index, serializedTask.limit(), maxRpcMessageSize)
taskSetMgr.abort(msg)
} catch {
case e: Exception => logError("Exception in error callback", e)
}
}
}
else {
val executorData = executorDataMap(task.executorId)
executorData.freeCores -= scheduler.CPUS_PER_TASK
logDebug(s"Launching task ${task.taskId} on executor id: ${task.executorId} hostname: " +
s"${executorData.executorHost}.")
// 向Executor发送LaunchTask消息
executorData.executorEndpoint.send(LaunchTask(new SerializableBuffer(serializedTask)))
}
}
CoarseGrainedSchedulerBackend.receive接收到消息:
case LaunchTask(data) =>
if (executor == null) {
exitExecutor(1, "Received LaunchTask command but executor was null")
} else {
val taskDesc = TaskDescription.decode(data.value)
logInfo("Got assigned task " + taskDesc.taskId)
executor.launchTask(this, taskDesc)
}
Executor的线程池执行任务,对应粉色6:
def launchTask(context: ExecutorBackend, taskDescription: TaskDescription): Unit = {
val tr = new TaskRunner(context, taskDescription)
runningTasks.put(taskDescription.taskId, tr)
threadPool.execute(tr)
}
至此,任务调度完毕。
任务执行与返回
Executor中线程池的某个线程调用run()方法执行task,完成后状态更新,对应绿色1
setTaskFinishedAndClearInterruptStatus()
execBackend.statusUpdate(taskId, TaskState.FINISHED, serializedResult)
跟进CoarseGrainedSchedulerBackend.statusUpdate方法,
override def statusUpdate(taskId: Long, state: TaskState, data: ByteBuffer) {
val msg = StatusUpdate(executorId, taskId, state, data)
driver match {
// 向DriverEndPoint发送StatusUpdate消息
case Some(driverRef) => driverRef.send(msg)
case None => logWarning(s"Drop $msg because has not yet connected to driver")
}
}
DriverEndPoint接收到StatusUpdate消息,执行绿色3和绿色4:
override def receive: PartialFunction[Any, Unit] = {
case StatusUpdate(executorId, taskId, state, data) =>
// 执行TaskSchedulerImpl.statusUpdate
scheduler.statusUpdate(taskId, state, data.value)
if (TaskState.isFinished(state)) {
executorDataMap.get(executorId) match {
case Some(executorInfo) =>
executorInfo.freeCores += scheduler.CPUS_PER_TASK
//回到makeOffers中,再次执行scheduler.resourceOffers()
makeOffers(executorId)
case None =>
// Ignoring the update since we don't know about the executor.
logWarning(s"Ignored task status update ($taskId state $state) " +
s"from unknown executor with ID $executorId")
}
}
总结
流程详解如下:
- 集群启动之后,Worker节点会向Master节点汇报资源情况,Master就掌握了集群资源情况。
- 当Spark提交一个Application后,会根据RDD之间的依赖关系将Application形成一个DAG有向无环图。任务提交之后,Spark会在Driver端创建两个对象:DAGScheduler和TaskScheduler
- DAGScheduler是任务调度的高层调度器,是一个对象。DAGScheduler的主要作用就是将DAG根据RDD之间的宽窄依赖关系划分为一个个的Stage,然后将这些Stage以TaskSet的形式提交给TaskScheduler(TaskScheduler是任务调度的低层调度器,这里TaskSet其实就是一个集合,里面封装的就是一个个的task任务,也就是stage中的并行度task任务)
- TaskSchedule会遍历TaskSet集合,拿到每个task后会将task发送到计算节点Executor中去执行(其实就是发送到Executor中的线程池ThreadPool去执行)。
- task在Executor线程池中的运行情况会向TaskScheduler反馈,当task执行失败时,则由TaskScheduler负责重试,将task重新发送给Executor去执行,默认重试3次。如果重试3次依然失败,那么这个task所在的stage就失败了。stage失败了则由DAGScheduler来负责重试,重新发送TaskSet到TaskSchdeuler**,Stage默认重试4次。**如果重试4次以后依然失败,那么这个job就失败了。job失败了,Application就失败了。因此一个task默认情况下重试3*4=12次。
- TaskScheduler不仅能重试失败的task,还会重试straggling(落后,缓慢)task(也就是执行速度比其他task慢太多的task)。如果有运行缓慢的task那么TaskScheduler会启动一个新的task来与这个运行缓慢的task执行相同的处理逻辑。两个task哪个先执行完,就以哪个task的执行结果为准。这就是Spark的推测执行机制。在Spark中推测执行默认是关闭的。推测执行可以通过spark.speculation属性来配置。
参考链接:
1.Spark 资源调度和任务调度源码解析
2. 深入理解Spark内核
3. Spark源码阅读: Spark Submit任务提交
4. standalone运行模式
5. Standalone模式运行机制