mac os x 编译spark-2.1.0 for hadoop-2.7.3

mac os x maven编译spark-2.1.0  for hadoop-2.7.3

 
1.官方文档中要求安装Maven 3.3.9+ 和Java 8 ;
 
2.执行     

   export MAVEN_OPTS="-Xmx2g -XX:ReservedCodeCacheSize=512m"

3.cd spark2.1.0源码根目录下
 ./build/mvn -Pyarn  -Phadoop-2.7  -Dhadoop.version=2.7.3  -Dscala-2.11      -Phive  -Phive-thriftserver    -DskipTests clean package

--

[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Spark Project Parent POM ........................... SUCCESS [02:10 min]
[INFO] Spark Project Tags ................................. SUCCESS [ 32.297 s]
[INFO] Spark Project Sketch ............................... SUCCESS [ 3.362 s]
[INFO] Spark Project Networking ........................... SUCCESS [ 22.433 s]
[INFO] Spark Project Shuffle Streaming Service ............ SUCCESS [ 6.130 s]
[INFO] Spark Project Unsafe ............................... SUCCESS [ 10.949 s]
[INFO] Spark Project Launcher ............................. SUCCESS [ 41.786 s]
[INFO] Spark Project Core ................................. SUCCESS [03:11 min]
[INFO] Spark Project ML Local Library ..................... SUCCESS [ 25.018 s]
[INFO] Spark Project GraphX ............................... SUCCESS [ 10.133 s]
[INFO] Spark Project Streaming ............................ SUCCESS [ 22.240 s]
[INFO] Spark Project Catalyst ............................. SUCCESS [01:05 min]
[INFO] Spark Project SQL .................................. SUCCESS [01:41 min]
[INFO] Spark Project ML Library ........................... SUCCESS [ 58.186 s]
[INFO] Spark Project Tools ................................ SUCCESS [ 5.925 s]
[INFO] Spark Project Hive ................................. SUCCESS [01:11 min]
[INFO] Spark Project REPL ................................. SUCCESS [ 3.290 s]
[INFO] Spark Project YARN Shuffle Service ................. SUCCESS [ 4.925 s]
[INFO] Spark Project YARN ................................. SUCCESS [ 20.450 s]
[INFO] Spark Project Hive Thrift Server ................... SUCCESS [ 19.625 s]
[INFO] Spark Project Assembly ............................. SUCCESS [ 2.035 s]
[INFO] Spark Project External Flume Sink .................. SUCCESS [ 15.382 s]
[INFO] Spark Project External Flume ....................... SUCCESS [ 6.557 s]
[INFO] Spark Project External Flume Assembly .............. SUCCESS [ 1.421 s]
[INFO] Spark Integration for Kafka 0.8 .................... SUCCESS [ 15.826 s]
[INFO] Spark Project Examples ............................. SUCCESS [ 14.344 s]
[INFO] Spark Project External Kafka Assembly .............. SUCCESS [ 2.765 s]
[INFO] Spark Integration for Kafka 0.10 ................... SUCCESS [ 13.367 s]
[INFO] Spark Integration for Kafka 0.10 Assembly .......... SUCCESS [ 2.907 s]
[INFO] Kafka 0.10 Source for Structured Streaming ......... SUCCESS [ 6.780 s]
[INFO] Spark Project Java 8 Tests ......................... SUCCESS [ 3.895 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 15:32 min
[INFO] Finished at: 2017-05-01T14:11:28+08:00
[INFO] Final Memory: 88M/981M
[INFO] ------------------------------------------------------------------------

--

[WARNING] The requested profile "hadoop-2.8" could not be activated because it does not exist.

mvn 从github 下载 hadoop-2.8.0 的lib文件，这些lib文件还没人准备。。。因此下载不到，不会编译。

 

4切换到编译完的dev目录下执行下面命令打包

  测试mvn编译成功后，可以用如下命令生成安装包：(当然直接执行生成安装包，但安全起见，还是先执测试一下)

./dev/make-distribution.sh --name custom-spark --tgz -Psparkr -Phadoop-2.7   -Phive -Phive-thriftserver  -Pyarn

参考：
http://spark.apache.org/docs/latest/building-spark.html

转载于:https://www.cnblogs.com/GrantYu/p/6791773.html