JobScheduler内幕实现和深度思考(第七篇)

最新推荐文章于 2024-08-11 00:11:40 发布

xiaojun220

最新推荐文章于 2024-08-11 00:11:40 发布

阅读量894

点赞数

分类专栏： spark大数据

本文链接：https://blog.csdn.net/xiaojun220/article/details/51475375

版权

本文深入探讨了Spark的JobScheduler内部实现，包括JobGenerator如何生成Job，jobScheduler如何提交Job给jobExecutor，以及JobHandler的关键逻辑。文章通过分析代码展示了Job的生命周期，从DStreamGraph.generateJobs到输出操作的foreachFunc，最后到实际业务逻辑的执行。同时，提到了动态调整jobExecutor并行度的优化策略，并引用了一张图来帮助理解整个流程。

摘要由CSDN通过智能技术生成

接着上一节我们看到了JobGenerator.generateJobs()方法：

private def generateJobs(time: Time) {
    // Set the SparkEnv in this thread, so that job generation code can access the environment
    // Example: BlockRDDs are created in this thread, and it needs to access BlockManager
    // Update: This is probably redundant after threadlocal stuff in SparkEnv has been removed.
    SparkEnv.set(ssc.env)
    Try {
      jobScheduler.receiverTracker.allocateBlocksToBatch(time) // allocate received blocks to batch
      graph.generateJobs(time) // generate jobs using allocated block
    } match {
      case Success(jobs) =>
        val streamIdToInputInfos = jobScheduler.inputInfoTracker.getInfo(time)
        jobScheduler.submitJobSet(JobSet(time, jobs, streamIdToInputInfos))
      case Failure(e) =>
        jobScheduler.reportError("Error generating jobs for time " + time, e)
    }
    eventLoop.post(DoCheckpoint(time, clearCheckpointDataLater = false))
  }