spark源码学习-错误汇总

在编译spark源码时遇到的一些问题整理一下

编译环境

系统：macOS Ventura 13.1
java：1.8.0_292
maven：3.8.1
spark：branch-3.3

Failed to execute goal org.apache.maven.plugins:maven-enforcer-plugin:3.0.0-M2:enforce

spark目录下执行：

mvn clean package  -Phive -Phive-thriftserver -Pyarn -DskipTests

发现报错：

[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  1.230 s
[INFO] Finished at: 2023-02-07T11:52:54+08:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-enforcer-plugin:3.0.0-M2:enforce (enforce-versions) on project spark-parent_2.12: Some Enforcer rules have failed. Look above for specific messages explaining why the rule failed. -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException

原因

本机Maven版本同spark工程中的Maven版本不同

解决方法

1.修改spark工程根目录下pom.xml文件中的 maven.version为本机版本

2.重装本机maven，版本和spark的相同

object SqlBaseParser is not a member of package org.apache.spark.sql.catalyst.parser

编译或者运行spark提供的测试类时，出现：

Error:(36, 45) object SqlBaseParser is not a member of package org.apache.spark.sql.catalyst.parser
import org.apache.spark.sql.catalyst.parser.SqlBaseParser._

原因

idea不会自动下载部分catalyst相关的源文件，导致编译时不能通过。

解决方法

在Intellij IDEA里面：

打开View -> Tool Windows -> Maven Projects
右击Spark Project Catalyst
点击Generate Sources and Update Folders
随后，Intellij IDEA会自动下载Catalyst相关的包
随后重新load 一下Spark Project Catalyst模块就可以了