spark
dab-hand
这个作者很懒,什么都没留下…
展开
-
Spark Job提交的整体流程源码详解
Job的提交任何一个行动算子,都会执行:runJob()val results = sc.runJob()runJob()在DAGScheduler类中1. 提交job,是一个阻塞线程 val waiter = submitJob(rdd, func, partitions, callSite, resultHandler, properties) //7422. submitJob中: val jobId = nextJobId.getAndIncrement() //生成JobId,原创 2020-09-20 16:33:29 · 209 阅读 · 0 评论 -
Spark Submit流程源码详解(on yarn)
Submit流程(yarn) #执行提交命令: bin/spark-submit --master yarn --deploy-mode cluster --class com.Tencent.spark.WordCount /opt/module/spark-standalone/bin/spark-1.0-SNAPSHOT.jar hdfs://hadoop:9820/input 执行脚本实际是执行: bin/spark-class org.apache.spark.dep原创 2020-09-20 16:30:56 · 372 阅读 · 0 评论