[Spark进阶]--再识spark高阶架构

最新推荐文章于 2024-07-11 23:54:02 发布

往事随风ing

最新推荐文章于 2024-07-11 23:54:02 发布

阅读量680

点赞数 2

分类专栏： Spark 文章标签： spark

本文链接：https://blog.csdn.net/high2011/article/details/83895232

版权

Spark 专栏收录该内容

133 篇文章 10 订阅 ¥9.90 ¥99.00

订阅专栏

超级会员免费看

本文详细介绍了Spark核心架构中的Driver Programs、Executors和Tasks。Driver Program作为应用主体，Spark在Worker上创建Executor并以JVM进程运行，Executor通过多线程并发执行任务。Task是Spark发送到Executor的最小工作单元，按数据分区创建。文章还阐述了Application Execution过程，包括Shuffle、Job和Stage的概念，以及Spark如何将作业拆分为DAG并在Executor上并行执行。

摘要由CSDN通过智能技术生成

Spark EcoSystem几乎都是以 Spark Core为核心而构建起来的，那么，先看看 Spark Core的高阶架构：

分别介绍下几个概念

1、Driver Programs
A driver program is an application that uses Spark as a library. It provides the data processing code that Spark executes on the worker nodes. A driver program can launch one or more jobs on a Spark cluster.

2、Executors
An executor is a JVM (Java virtual machine) process that Spark creates on each worker for an application. It executes application code concurrently in multiple threads. It can also cache data in memory or disk.
An executor has

了解本专栏