Spark源码编译

[hadoop@node1 dev]$ ./make-distribution.sh --name 2.6.0-cdh5.7.0 --tgz -Pyarn -Phadoop-2.6 -Phive -Phive-thriftserver -Dhadoop.version=2.6.0-cdh5.7.0

坑一 会报如下的错误

[ERROR] Failed to execute goal on project spark-launcher_2.11: Could not resolve dependencies for project org.apache.spark:spark-launcher_2.11:jar:2.2.0: Could not find artifact org.apache.hadoop:hadoop-client:jar:2.6.0-cdh5.7.0 in nexus-aliyun (http://maven.aliyun.com/nexus/content/groups/public) -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException

编辑 pom.xml 文件 ，添加如下的内容 蓝色字体需要添加的， 默认就行了

</repositories>
<repository>
     <id>cloudera</id>
    <name>cloudera repository</name>
    <url>https://repository.cloudera.com/artifactory/cloudera-repos/</url>
    </repository> 
<repositories>

有条件的话 推荐VPN(可以访问谷歌等网站)

坑二 ： 有时编译的机器内存不够
注意编辑机器的内存 一般推荐4G以上的内存

export MAVEN_OPTS="-Xmx2g -XX:ReservedCodeCacheSize=512m"

坑三：
如果编译的是Scala版本是2.10 需要先执行如下的脚本

./dev/change-scala-version.sh 2.10

编译完成后 会打包生成一个tar包

spark-2.2.0-bin-2.6.0-cdh5.7.0.tgz

坑四：编译过程由于网络问题卡住
更改make-distribution.sh里面的$MAVEN_HOME，使用自己的maven设置进行下载

MVN="$MAVEN_HOME/bin/mvn"