![v2-cb052a09c391bd0d33946fe86ef7815d_1440w.jpg?source=172ae18b](http://img-02.proxy.5ce.com/view/image?&type=2&guid=9fa61176-3f2f-eb11-8da9-e4434bdf6706&url=https://pic2.zhimg.com/v2-cb052a09c391bd0d33946fe86ef7815d_1440w.jpg?source=172ae18b)
注:图片如果损坏,点击文章链接:Spark源码编译
Spark官网下载地址:
http://spark.apache.org/downloads.html
有两种一种是编译好的 一种是没编译的(需要自己编译) 看自己的选择
![v2-abd620dffd3eae1a86474746ef1dcbff_b.jpg](http://img-03.proxy.5ce.com/view/image?&type=2&guid=9fa61176-3f2f-eb11-8da9-e4434bdf6706&url=https://pic4.zhimg.com/v2-abd620dffd3eae1a86474746ef1dcbff_b.jpg)
下载好的文件
![v2-13791801969bf95bf216245eb5d89b7e_b.jpg](http://img-02.proxy.5ce.com/view/image?&type=2&guid=9fa61176-3f2f-eb11-8da9-e4434bdf6706&url=https://pic3.zhimg.com/v2-13791801969bf95bf216245eb5d89b7e_b.jpg)
解压之后可以看源码
![v2-73725ceb9c04e9ba4e43b4db1a765609_b.png](http://img-03.proxy.5ce.com/view/image?&type=2&guid=9fa61176-3f2f-eb11-8da9-e4434bdf6706&url=https://pic2.zhimg.com/v2-73725ceb9c04e9ba4e43b4db1a765609_b.png)
直接用IDE工具打开就可以
![v2-e5690f1edc6ee762c16a6b741d1df8a4_b.jpg](http://img-02.proxy.5ce.com/view/image?&type=2&guid=9fa61176-3f2f-eb11-8da9-e4434bdf6706&url=https://pic1.zhimg.com/v2-e5690f1edc6ee762c16a6b741d1df8a4_b.jpg)
从官网下载需要的版本https://archive.apache.org/dist/spark/
![v2-13a320a21312b152f0178a0f2e57485f_b.jpg](http://img-03.proxy.5ce.com/view/image?&type=2&guid=9fa61176-3f2f-eb11-8da9-e4434bdf6706&url=https://pic4.zhimg.com/v2-13a320a21312b152f0178a0f2e57485f_b.jpg)
![v2-c40f97288f6e230e5357d69406aa63a5_b.jpg](http://img-03.proxy.5ce.com/view/image?&type=2&guid=9fa61176-3f2f-eb11-8da9-e4434bdf6706&url=https://pic2.zhimg.com/v2-c40f97288f6e230e5357d69406aa63a5_b.jpg)
解压文件
![v2-cb3cd5f806f6ac1a083215d928067763_b.png](http://img-03.proxy.5ce.com/view/image?&type=2&guid=9fa61176-3f2f-eb11-8da9-e4434bdf6706&url=https://pic4.zhimg.com/v2-cb3cd5f806f6ac1a083215d928067763_b.png)
编译前的环境准备
Maven 3.3.9+
![v2-b55c91c089ce5c69415c69a29469f9ad_b.jpg](http://img-03.proxy.5ce.com/view/image?&type=2&guid=9fa61176-3f2f-eb11-8da9-e4434bdf6706&url=https://pic2.zhimg.com/v2-b55c91c089ce5c69415c69a29469f9ad_b.jpg)
jdk7以上
![v2-e22849556d41687f05767745d943f896_b.jpg](http://img-02.proxy.5ce.com/view/image?&type=2&guid=9fa61176-3f2f-eb11-8da9-e4434bdf6706&url=https://pic3.zhimg.com/v2-e22849556d41687f05767745d943f896_b.jpg)
scala 2.11+
![v2-13c58e4c4de7c1dcf88fe6b01fd47780_b.png](http://img-02.proxy.5ce.com/view/image?&type=2&guid=9fa61176-3f2f-eb11-8da9-e4434bdf6706&url=https://pic1.zhimg.com/v2-13c58e4c4de7c1dcf88fe6b01fd47780_b.png)
注意:在配置java之前,需要检查以下Linux是否有自带jdk,如果有的话,先删除,然后再解压jdk进行配置
用以下命令去检查
rpm -qa | grep java
这里是我自己下载的就不删除了
![v2-7092b19470cc8390c70bc554318047a8_b.jpg](http://img-03.proxy.5ce.com/view/image?&type=2&guid=9fa61176-3f2f-eb11-8da9-e4434bdf6706&url=https://pic1.zhimg.com/v2-7092b19470cc8390c70bc554318047a8_b.jpg)
卸载: rpm -e --nodeps xxxxxx aaaaa ddddd
在环境变量中配置以下三个环境变量
#JAVA_HOME
export JAVA_HOME=/opt/modules/jdk1.8.0_11
export PATH=$PATH:$JAVA_HOME/bin
#MAVEN_HOME
export MAVEN_HOME=/opt/modules/apache-maven-3.3.9
export PATH=$PATH:$MAVEN_HOME/bin
#SCALA_HOME
export SCALA_HOME=/opt/modules/scala-2.11.8
export PATH=$PATH:$SCALA_HOME/bin
使用shell命令进行编译(spark源码主目录下)执行
./dev/make-distribution.sh --name 2.7.3 --tgz -Phadoop-2.7 -Dhadoop.version=2.7.3 -Phive -Phive-thriftserver -Pyarn -DskipTests-Dscala-2.11
-Pyarn:表示支持yarn
-Phadoop-2.7:表示支持hadoop的大版本
-Dhadoop.version=2.6.0-cdh5.7.0:具体的hadoop版本
-Phive -Phive-thriftserver :支持hive和hive-thrifserver
-Dscala-2.11:表示对应的scala版本
![v2-4a62d7a2216e5e8ef3df3b191ed94a4a_b.png](http://img-01.proxy.5ce.com/view/image?&type=2&guid=9fa61176-3f2f-eb11-8da9-e4434bdf6706&url=https://pic3.zhimg.com/v2-4a62d7a2216e5e8ef3df3b191ed94a4a_b.png)
![v2-ddab9957fdf85a872dcfbe559a752be3_b.jpg](http://img-02.proxy.5ce.com/view/image?&type=2&guid=9fa61176-3f2f-eb11-8da9-e4434bdf6706&url=https://pic4.zhimg.com/v2-ddab9957fdf85a872dcfbe559a752be3_b.jpg)
需要等一段时间
注意事项
虚拟机内存最好4G以上,重启虚拟机再编译
编译失败:网络不好、内存不够
如果编译cdh版本,需要改一下仓库url
<repository>
<id>cloudera</id>
<name>cloudera Repository</name>
<url>https://repository.cloudera.com/artifactory/cloudera-repos</url>
</repository>
这样就成功了
![v2-9660bc07555bf0e1e3236fb362a7d781_b.jpg](http://img-02.proxy.5ce.com/view/image?&type=2&guid=9fa61176-3f2f-eb11-8da9-e4434bdf6706&url=https://pic2.zhimg.com/v2-9660bc07555bf0e1e3236fb362a7d781_b.jpg)
编译就完成了
![v2-7ef8a7f8ac7e813e5adc0aba4afbd963_b.jpg](http://img-01.proxy.5ce.com/view/image?&type=2&guid=9fa61176-3f2f-eb11-8da9-e4434bdf6706&url=https://pic4.zhimg.com/v2-7ef8a7f8ac7e813e5adc0aba4afbd963_b.jpg)