【Spark】Spark Web UI - Jobs

  工作中经常会出现 Spark SQL 执行很慢或者失败的情况，如果要排查问题，就必须要学会看 Spark Web UI。可以参考官网来学习：https://spark.apache.org/docs/3.2.1/web-ui.html#content。关于 Spark Web UI，上面有很多个 tab 页，今天开始逐一学习。

首先是 Jobs。

Jobs Tab

The Jobs tab displays a summary page of all jobs in the Spark application and a details page for each job. The summary page shows high-level information, such as the status, duration, and progress of all jobs and the overall event timeline. When you click on a job on the summary page, you see the details page for that job. The details page further shows the event timeline, DAG visualization, and all stages of the job.

作业选项卡显示 Spark 应用程序中所有作业的摘要页面以及每个作业的详细信息页面。 摘要页面显示高级信息，例如所有作业的状态、持续时间和进度以及整个事件时间线。 当您单击摘要页面上的作业时，您会看到该作业的详细信息页面。 详细信息页面进一步显示了事件时间线、DAG 可视化和作业的所有阶段。

The information that is displayed in this section is

• User: Current Spark user
• Total uptime: Time since Spark application started
• Scheduling mode: See job scheduling
• Number of jobs per status: Active, Completed, Failed
• Event timeline: Displays in chronological order the events related to the executors (added, removed) and the jobs
• Details of jobs grouped by status: Displays detailed information of the jobs including Job ID, description (with a link to detailed job page), submitted time, duration, stages summary and tasks progress bar

此部分中显示的信息是：

• 用户：当前 Spark 用户
• 总正常运行时间：自 Spark 应用程序启动以来的时间
• 调度模式：见作业调度
• 每个状态的作业数：活动、已完成、失败

• 事件时间线：按时间顺序显示与 executor（添加、删除）和作业相关的事件

• 按状态分组的作业详细信息：显示作业的详细信息，包括作业 ID、描述（带有详细作业页面的链接）、提交时间、持续时间、阶段摘要和任务进度条

单击特定作业时，您可以看到该作业的详细信息。

Jobs detail

This page displays the details of a specific job identified by its job ID.

• Job Status: (running, succeeded, failed)
• Number of stages per status (active, pending, completed, skipped, failed)
• Associated SQL Query: Link to the sql tab for this job
• Event timeline: Displays in chronological order the events related to the executors (added, removed) and the stages of the job
• DAG visualization: Visual representation of the directed acyclic graph of this job where vertices represent the RDDs or DataFrames and the edges represent an operation to be applied on RDD.
• An example of DAG visualization for sc.parallelize(1 to 100).toDF.count()
• List of stages (grouped by state active, pending, completed, skipped, and failed)
  • Stage ID
  • Description of the stage
  • Submitted timestamp
  • Duration of the stage
  • Tasks progress bar
  • Input: Bytes read from storage in this stage
  • Output: Bytes written in storage in this stage
  • Shuffle read: Total shuffle bytes and records read, includes both data read locally and data read from remote executors
  • Shuffle write: Bytes and records written to disk in order to be read by a shuffle in a future stage

此页面显示由其作业 ID 标识的特定作业的详细信息。

• 作业状态：正在运行、成功、失败
• 每个状态的阶段数：活动、待处理、已完成、已跳过、失败
• 关联 SQL 查询：链接到此作业的 sql 选项卡
• 事件时间线：按时间顺序显示与该作业相关的事件
• 执行者（添加、删除）和作业的阶段

• DAG 可视化：此作业的有向无环图的可视化表示，其中顶点表示 RDD 或 DataFrame，边表示要应用于 RDD 的操作。
• DAG 可视化示例：sc.parallelize(1 to 100).toDF.count()

• 阶段列表（按状态活动、未决、完成、跳过和失败分组）
  • 阶段 ID
  • 阶段描述
  • 提交的时间戳
  • 阶段的持续时间
  • 任务进度条
  • 输入：在此阶段从存储读取的字节数
  • 输出：在此阶段的存储中写入的字节数
  • Shuffle read：总的 shuffle 字节和读取的记录，包括本地读取的数据和从远程执行器读取的数据
  • Shuffle write：写入磁盘的字节和记录，以便在未来阶段被 shuffle 读取

欢迎点击此处关注公众号。