SparkControlProcesses
Driver | Application entry point that contains the SparkContext instance |
Master | In charge of scheduling and resource orchestration |
Worker | Responsible for node state and running executors A worker is a control process that runs on a cluster node |
Executor | Allocated per job and in charge of executing tasks from that job An executor is a JVM process that executes on each worker |
SparkExecutionHierarchy
Application | One instance of a SparkContext |
Job | Set of stages executed as a result of an action |
Stage | Set of transformations at a shuffle boundary |
Task set | Set of tasks from the same stage |
Task | Unit of execution in a stage Tasks in Spark are the unit of execution. Tasks execute as threads instead of processes in the executor, to enable in-memory RDDs. By default, there is a one-to-one mapping between a core and a task. |
job、stage、task
- Worker Node:物理节点,上面执行executor进程
- Executor:Worker Node为某应用启动的一个进程,执行多个tasks
- Jobs:action 的触发会生成一个job, Job会提交给DAGScheduler,分解成Stage,
- Stage:DAGScheduler 根据shuffle将job划分为不同的stage,同一个stage中包含多个task,这些tasks有相同的 shuffle dependencies。
有两类shuffle map stage和result stage:
shuffle map stage:case its tasks' results are input for other stage(s)
result stage:case its tasks directly compute a Spark action (e.g. count(), save(), etc) by running a function on an RDD,输入与结果间划分stage
- Task:被送到executor上的工作单元,task简单的说就是在一个数据partition上的单个数据处理流程。
小结:
action触发一个job (task对应在一个partition上的数据处理流程)
------stage1(多个tasks 有相同的shuffle依赖)------【map--shuffle】------- stage2---- 【result--shuffle】-----
【样例说明】
已下面的一段样例分析下具体的执行过程,为了分析,将map和reduce没有写成链式
data.txt数据如下,分别表示event uuid pv
1 1001 0
1 1001 1
1 1002 0
1 1003 1
2 1002 1
2 1003 1
2 1003 0
3 1001 0
3 1001 0
计算每个事件的uv和pv,即实现
event维度下,count(distinct if(pv > 0,uuid,null)) ,sum(pv)
-
import org.apache.spark.{SparkConf, SparkContext}
-
-
/**
-
* Created by hjw on 17/9/11.
-
*
-
**/
-
/*
-
event uuid pv data.txt的数据
-
1 1001 0
-
1 1001 1
-
1 1002 0
-
1 1003 1
-
2 1002 1
-
2 1003 1
-
2 1003 0
-
3 1001 0
-
3 1001 0
-
计算 event,count(distinct if(pv > 0,uuid,null)) ,sum(pv)
-
2 UV=2 PV=2
-
3 UV=0 PV=0
-
1 UV=2 PV=2
-
为了分析,将map和reduce没有写成链式
-
*/
-
object DAG {
-
-
def main(args: Array[String]) {
-
val conf =
new SparkConf()
-
-
conf.setAppName(
"test")
-
conf.setMaster(
"local")
-
-
val sc =
new SparkContext(conf)
-
-
val txtFile = sc.textFile(
".
xxxxxx/
DAG/srcFile/data.txt")
-
val inputRDD = txtFile.map(x => (x.split(
"\t")(
0), x.split(
"\t")(
1), x.split(
"\t")(
2).toInt))
-
//val partitionsSzie = inputRDD.partitions.size
-
-
//这里为了分析task数先重分区,分区前partitions.size = 1,下面每个stage的task数为1
-
val inputPartionRDD = inputRDD.repartition(
2)
-
-
//------map_shuffle stage 有shuffle Read
-
//结果:(事件-用户,pv)
-
val eventUser2PV = inputPartionRDD.map(x => (x._1 +
"-" + x._2, x._3))
-
-
//结果: (事件,(用户,pv))
-
val PvRDDTemp1 = eventUser2PV.reduceByKey(_ + _).map(x =>
-
(x._1.split(
"-")(
0), (x._1.split(
"-")(
1), x._2))
-
)
-
-
//-------map_shuffle stage 有shuffle Read 和 有shuffle Write
-
//结果: (事件, Tuple2(Tuple2(用户,是否出现),该用户的pv) )
-
val PvUvRDDTemp2 = PvRDDTemp1.map(
-
x => x match {
-
case x
if x._2._2 >
0 => (x._1, (
1, x._2._2))
-
case x
if x._2._2 ==
0 => (x._1, (
0, x._2._2))
-
}
-
)
-
-
//结果:(事件,Tuple2(uv,pv))
-
val PVUVRDD = PvUvRDDTemp2.reduceByKey(
-
(a, b) => (a._1 + b._1, a._2 + b._2)
-
)
-
-
//------result_shuffle stage 有shuffle Read
-
//--------触发一个job
-
val res = PVUVRDD.collect();
-
-
-
//------result_shuffle stage 有shuffle Read
-
//--------触发一个job
-
PVUVRDD.foreach(a => println(a._1 +
"\t UV=" + a._2._1 +
"\t PV=" + a._2._2))
-
// 2 UV=2 PV=2
-
// 3 UV=0 PV=0
-
// 1 UV=2 PV=2
-
while (
true) {
-
;
-
}
-
sc.stop()
-
}
-
}
日志中拣选出关键分析点如下:
-
17/09/11 22:46:19 INFO SparkContext: Running Spark version 1.6.
-
17/
09/
11
22:
46:
21 INFO SparkUI: Started SparkUI at http:
//192.168.2.100:4040
-
-
17/
09/
11
22:
46:
21 INFO FileInputFormat: Total input paths to process :
1
-
17/
09/
11
22:
46:
22 INFO SparkContext: Starting job: collect at DAG.scala:
69
-
17/
09/
11
22:
46:
22 INFO DAGScheduler: Registering RDD
3 (repartition at DAG.scala:
42)
-
17/
09/
11
22:
46:
22 INFO DAGScheduler: Registering RDD
7 (map at DAG.scala:
46)
-
17/
09/
11
22:
46:
22 INFO DAGScheduler: Registering RDD
10 (map at DAG.scala:
55)
-
-
17/
09/
11
22:
46:
22 INFO DAGScheduler: Got job
0 (collect at DAG.scala:
69) with
2 output partitions
-
17/
09/
11
22:
46:
22 INFO DAGScheduler: Final stage: ResultStage
3 (collect at DAG.scala:
69)
-
-
-
17/
09/
11
22:
46:
22 INFO DAGScheduler: Submitting ShuffleMapStage
0 (MapPartitionsRDD[
3] at repartition at DAG.scala:
42), which has no missing parents
-
17/
09/
11
22:
46:
22 INFO DAGScheduler: ShuffleMapStage
0 (repartition at DAG.scala:
42) finished in
0.106 s
-
17/
09/
11
22:
46:
22 INFO DAGScheduler: waiting: Set(ShuffleMapStage
1, ShuffleMapStage
2, ResultStage
3)
-
-
17/
09/
11
22:
46:
22 INFO DAGScheduler: Submitting ShuffleMapStage
1 (MapPartitionsRDD[
7] at map at DAG.scala:
46), which has no missing parents
-
17/
09/
11
22:
46:
22 INFO DAGScheduler: Submitting
2 missing tasks from ShuffleMapStage
1 (MapPartitionsRDD[
7] at map at DAG.scala:
46)
-
17/
09/
11
22:
46:
22 INFO DAGScheduler: ShuffleMapStage
1 (map at DAG.scala:
46) finished in
0.055 s
-
-
17/
09/
11
22:
46:
22 INFO DAGScheduler: waiting: Set(ShuffleMapStage
2, ResultStage
3)
-
17/
09/
11
22:
46:
22 INFO DAGScheduler: Submitting ShuffleMapStage
2 (MapPartitionsRDD[
10] at map at DAG.scala:
55), which has no missing parents
-
17/
09/
11
22:
46:
22 INFO DAGScheduler: ShuffleMapStage
2 (map at DAG.scala:
55) finished in
0.023 s
-
-
17/
09/
11
22:
46:
22 INFO DAGScheduler: Submitting ResultStage
3 (ShuffledRDD[
11] at reduceByKey at DAG.scala:
63), which has no missing parents
-
17/
09/
11
22:
46:
22 INFO DAGScheduler: ResultStage
3 (collect at DAG.scala:
69) finished in
0.009 s
-
17/
09/
11
22:
46:
22 INFO DAGScheduler: Job
0 finished: collect at DAG.scala:
69, took
0.283076 s
-
-
-
17/
09/
11
22:
46:
22 INFO SparkContext: Starting job: foreach at DAG.scala:
74
-
17/
09/
11
22:
46:
22 INFO DAGScheduler: Got job
1 (foreach at DAG.scala:
74) with
2 output partitions
-
-
17/
09/
11
22:
46:
22 INFO DAGScheduler: Final stage: ResultStage
7 (foreach at DAG.scala:
74)
-
17/
09/
11
22:
46:
22 INFO DAGScheduler: Parents of
final stage: List(ShuffleMapStage
6)
-
17/
09/
11
22:
46:
22 INFO DAGScheduler: Submitting ResultStage
7 (ShuffledRDD[
11] at reduceByKey at DAG.scala:
63), which has no missing parents
-
17/
09/
11
22:
46:
22 INFO DAGScheduler: ResultStage
7 (foreach at DAG.scala:
74) finished in
0.010 s
-
17/
09/
11
22:
46:
22 INFO DAGScheduler: Job
1 finished: foreach at DAG.scala:
74, took
0.028036 s
-
line 69 collect() 和 line 74 行 foreach()触发两个job
val res = PVUVRDD.collect();
PVUVRDD.foreach(a => println(a._1 + "\t UV=" + a._2._1 + "\t PV=" + a._2._2))
基于下图知
job0有4个stage,共7个task
- stage0:1个分区有1个task执行
- stage1--3:各有2个分区,各有2个task执行,共6个task
具体job0的细节
具体job0-stage0
1 1001 0
1 1001 1
1 1002 0
1 1003 1
2 1002 1
2 1003 1
2 1003 0
3 1001 0
3 1001 0
输入9条记录,只要一个分区,有一个task执行,想repartion进行shuffle写数据
执行过程分析:
Action触发Job,如从(1)中的collect 动作val res = PVUVRDD.collect(),开始逆向分析job执行过程
Action中利用SparkContext runJob()调用–dagScheduler.runJob(rdd,func,分区数,其他)提交Job作业
DAGScheduler的runJob中调用submitJob()并返回监听waiter,生命周期内监听Job状态
runJob()内部
val waiter = submitJob(rdd, func, partitions, callSite, resultHandler, properties),监听waiter(存在原子jobId)的状态也即Job是否完成
submitJob()内部
返回值waiter是JobWaiter(JobListener)
val waiter = new JobWaiter(this, jobId, partitions.size, resultHandler)
将该获取到的Job(已有JobId),插入到LinkedBlockingDeque结构的事件处理队列中eventProcessLoop
eventProcessLoop.post(JobSubmitted(
jobId, rdd, func2, partitions.toArray, callSite, waiter,
SerializationUtils.clone(properties)))
eventProcessLoop = new DAGSchedulerEventProcessLoop(this)
class DAGSchedulerEventProcessLoop(dagScheduler: DAGScheduler)
extends EventLoop
EventLoop是LinkedBlockingDeque
eventProcessLoop(LinkedBlockingDeque类型)放入新事件后,调起底层的DAGSchedulerEventProcessLoop.onReceive(),执行doOnReceive()
doOnReceive(event: DAGSchedulerEvent)内部
根据DAGSchedulerEvent的具体类型如JobSubmitted事件或者MapStageSubmitted事件,调取具体的Submitted handle函数提交具体的Job
如 event case JobSubmitted=>dagScheduler.handleJobSubmitted(jobId, rdd, func, partitions, callSite, listener, properties)
private def doOnReceive(event: DAGSchedulerEvent): Unit = event match {
case JobSubmitted(jobId, rdd, func, partitions, callSite, listener, properties) =>
dagScheduler.handleJobSubmitted(jobId, rdd, func, partitions, callSite, listener, properties)
case MapStageSubmitted(jobId, dependency, callSite, listener, properties) =>
dagScheduler.handleMapStageSubmitted(jobId, dependency, callSite, listener, properties)
handleJobSubmitted()内部
返回从ResultStage 建立stage 建立finalStage = createResultStage(finalRDD, func, partitions, jobId, callSite)
finalStage激活Job val job = new ActiveJob(jobId, finalStage, callSite, listener, properties),同时开始逆向构建缺失的stage,getMissingParentStages(finalStage)(待补充看)
DAG构建完毕,提交stage,submitStage(finalStage)
submitStage中stage提交为tasks,submitMissingTasks()
submitMissingTasks,根据ShuffleMapStage还是ResultStage--new ShuffleMapTask 或 ResultTask
taskScheduler.submitTasks()开始调起具体的task
private def submitStage(stage: Stage) {
val missing = getMissingParentStages(stage).sortBy(_.id)//====在这里开始反向划分,具体划分见下方
logDebug("missing: " + missing)
if (missing.isEmpty) {
logInfo("Submitting " + stage + " (" + stage.rdd + "), which has no missing parents")
submitMissingTasks(stage, jobId.get)
private def getMissingParentStages(stage: Stage): List[Stage] ={}
![](https://i-blog.csdnimg.cn/blog_migrate/d4b9c99e890a148911d846b58758521b.png)
taskScheduler.submitTasks 待续。。。
从Job逆向分析TaskScheduler工作原理
DAGScheduler将stage提交task,然后stage逆向执行该stage的中rdd,这些rdds被打包成tasksets,也即每个partition为一个task,在rdd上执行对应的function函数。
细节如下:
接上部分最后一部分,DAGScheduler中的submitStage-submitMissingTasks(stage, jobId.get)如下,提交每个stage
private def submitStage(stage: Stage) {
val missing = getMissingParentStages(stage).sortBy(_.id)//====在这里开始反向划分,具体划分见下方
submitMissingTasks()按照stage的类型,调用不同的stageStart的重载类型
待续。。