Spark2.4.0源码编译支持hadoop-2.6.0-cdh5.7.0

软件包

• JDK1.8.0_45
• apache-maven-3.6.0
• Scala2.11.8
• hadoop-2.6.0-cdh5.7.0

---

注意设置内存*****

export MAVEN_OPTS="-Xmx2g -XX:ReservedCodeCacheSize=512m"

---

安装git,源码编译过程中会需要用到git

[hadoop001@iZuf6dz84cmy6o2y05pb8bZ scala-2.11.8]$ sudo yum install -y git

---

下载spark2.4.0源码

[hadoop001@iZuf6dz84cmy6o2y05pb8bZ software]$ wget https://archive.apache.org/dist/spark/spark-2.4.0/spark-2.4.0.tgz
--2019-01-03 07:05:31--  https://archive.apache.org/dist/spark/spark-2.4.0/spark-2.4.0.tgz
Resolving archive.apache.org... 163.172.17.199
Connecting to archive.apache.org|163.172.17.199|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 16118412 (15M) [application/x-gzip]
Saving to: “spark-2.4.0.tgz”

100%[===============================================================================================================>] 16,118,412  3.17M/s   in 4.9s    

2019-01-03 07:05:37 (3.17 MB/s) - “spark-2.4.0.tgz” saved [16118412/16118412]

解压spark源码

[hadoop001@iZuf6dz84cmy6o2y05pb8bZ software]$ tar -zxvf spark-2.4.0.tgz -C ~/app/

• 因为使用的是CDH版本的Hadoop，所以需要在pom.xml添加repository

<repository>
     <id>cloudera</id>
     <url>https://repository.cloudera.com/artifactory/cloudera-repos/</url>
</repository>

编译方法

用make-distributed 脚本来编译

• 修改dev目录下make-distribution.sh脚本

VERSION=2.4.0        #Spark版本
SCALA_VERSION=2.11   #Scala版本（大版本）
SPARK_HADOOP_VERSION=2.6.0-cdh5.7.0 #Hadoop版本
SPARK_HIVE=1         #是否将Hive打包（非1表示不打包）

• 图示
  

• 执行编译命令

[hadoop001@iZuf6dz84cmy6o2y05pb8bZ spark-2.4.0]$ ./dev/make-distribution.sh --name 2.6.0-cdh5.7.0 --tgz -Pyarn -Phadoop-2.6 -Phive -Phive-thriftserver -Dhadoop.version=2.6.0-cdh5.7.0

耐心等待编译完成即可，编成功后生成的部署包位于根目录下。