1.下载
2.设置CDH附加仓库地址 和镜像加速
<repository>
<id>cloudera</id>
<url>https://repository.cloudera.com/artifactory/cloudera-repos/</url>
</repository>
<mirror>
<id>alimaven</id>
<name>aliyun maven</name>
<url>http://maven.aliyun.com/nexus/content/groups/public/</url>
<mirrorOf>central</mirrorOf>
</mirror>
3.编译
3.1.环境准备
vim /etc/profile.d/maven.sh
MAVEN_HOME=/usr/local/maven
export PATH=$MAVEN_HOME/bin:$PATH
3.2.[非必要] 根据需求修改scala版本
Spark-2.4.2 默认 scala版本是2.12
./dev/change-scala-version.sh 2.12
./dev/change-scala-version.sh 2.11
3.3.源码编译
./dev/make-distribution.sh --name 2.6.0-cdh5.13.0 --tgz -Phadoop-2.6 -Phive -Phive-thriftserver -Pyarn -Dhadoop.version=2.6.0-cdh5.13.0
./dev/make-distribution.sh --name 2.6.0-cdh5.13.0 --pip --r --tgz -Psparkr -Phadoop-2.6 -Dhadoop.version=2.6.0-cdh5.13.0 -Phive -Phive-thriftserver -Pyarn -Pmesos -Pkubernetes
编译结束后会看pom.xml所在的目录有对应的jar
4.问题解决
4.1scala版本错配
java.io.InvalidClassException:
org.apache.commons.lang3.time.FastDateParser; local class incompatible:
stream classdesc serialVersionUID = 2, local class serialVersionUID = 3
问题:编译时scala版本和部署环境scala版本不一致 解决:保持二者版本一致
5.参考文档
http://www.louisvv.com/archives/1775.html