spark源码下载及编译

从http://spark.apache.org/下载spark源码包。

然后

(1)解压

$tar -zxvf  spark-x.y.z-bin-hadoopx.y.tgz

(2)修改pom.xml文件

        该文件在解压目录中的conf文件夹当中，我的部分修改如下：

<java.version>1.8</java.version>
<hadoop.version>2.7.3</hadoop.version>
<protobuf.version>2.5.0</protobuf.version>
<hbase.version>0.98.9-hadoop2</hbase.version>
<zookeeper.version>3.4.6</zookeeper.version>
<derby.version>10.11.1.1</derby.version>

(3)如果要支持Scala 2.11，请运行以下脚本：-- 该版本太新了，目前还有些组件不支持，所以不建议修改

~spark-2.1.0$sh ./dev/change-version-to-2.11.sh

(4)修改maven环境变量配置

export MAVEN_OPTS="-Xmx4g -XX:MaxPermSize=1024M -XX:ReservedCodeCacheSize=1024m"

(5)maven编译

~spark-2.1.0$./build/mvn -Pyarn -Phadoop-2.7 -Dhadoop.version=2.7.3 -DskipTests clean package

该过程需要很长时间，也可能会出现build failure，多执行几次。

(6)生成安装包

   如果不需要hive，则执行

./dev/make-distribution.sh --name "hadoop2-without-hive" --tgz "-Pyarn,hadoop-provided,hadoop-2.7,parquet-provided"




如果需要hive，则执行

./dev/make-distribution.sh --name custom-spark --tgz -Psparkr -Phadoop-2.7 -Phive -Phive-thriftserver -Pmesos -Pyarn

我的没有要hive，最后编译成功信息如下：

main:
[INFO] Executed tasks
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO] 
[INFO] Spark Project Parent POM ........................... SUCCESS [ 19.393 s]
[INFO] Spark Project Tags ................................. SUCCESS [ 18.437 s]
[INFO] Spark Project Sketch ............................... SUCCESS [ 15.209 s]
[INFO] Spark Project Networking ........................... SUCCESS [ 19.325 s]
[INFO] Spark Project Shuffle Streaming Service ............ SUCCESS [ 12.823 s]
[INFO] Spark Project Unsafe ............................... SUCCESS [ 24.586 s]
[INFO] Spark Project Launcher ............................. SUCCESS [ 23.378 s]
[INFO] Spark Project Core ................................. SUCCESS [31:01 min]
[INFO] Spark Project ML Local Library ..................... SUCCESS [ 39.471 s]
[INFO] Spark Project GraphX ............................... SUCCESS [32:14 min]
[INFO] Spark Project Streaming ............................ SUCCESS [33:25 min]
[INFO] Spark Project Catalyst ............................. SUCCESS [  01:54 h]
[INFO] Spark Project SQL .................................. SUCCESS [  04:23 h]
[INFO] Spark Project ML Library ........................... SUCCESS [  01:08 h]
[INFO] Spark Project Tools ................................ SUCCESS [ 10.568 s]
[INFO] Spark Project Hive ................................. SUCCESS [25:59 min]
[INFO] Spark Project REPL ................................. SUCCESS [18:52 min]
[INFO] Spark Project YARN Shuffle Service ................. SUCCESS [ 42.815 s]
[INFO] Spark Project YARN ................................. SUCCESS [  01:08 h]
[INFO] Spark Project Assembly ............................. SUCCESS [ 35.475 s]
[INFO] Spark Project External Flume Sink .................. SUCCESS [ 25.452 s]
[INFO] Spark Project External Flume ....................... SUCCESS [27:54 min]
[INFO] Spark Project External Flume Assembly .............. SUCCESS [ 40.121 s]
[INFO] Spark Integration for Kafka 0.8 .................... SUCCESS [10:36 min]
[INFO] Spark Project Examples ............................. SUCCESS [06:14 min]
[INFO] Spark Project External Kafka Assembly .............. SUCCESS [ 34.261 s]
[INFO] Spark Integration for Kafka 0.10 ................... SUCCESS [40:25 min]
[INFO] Spark Integration for Kafka 0.10 Assembly .......... SUCCESS [ 15.937 s]
[INFO] Kafka 0.10 Source for Structured Streaming ......... SUCCESS [18:20 min]
[INFO] Spark Project Java 8 Tests ......................... SUCCESS [02:35 min]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 12:49 h
[INFO] Finished at: 2017-04-26T21:07:50+08:00
[INFO] Final Memory: 126M/904M
[INFO] ------------------------------------------------------------------------

可以看到花的时间非常久。编译成功后的安装包是在最早的解压目录下，文件名是spark-2.1.0-bin-hadoop2-without-hive.tgz