spark2.4.2-cdh5.7.0源码编译

spark2.4.2-cdh5.7.0源码编译

参考文档

• spark github 源码 https://github.com/apache/spark/tree/v2.4.2
• 编译spark环境介绍 http://spark.apache.org/docs/latest/building-spark.html
• 编译脚本 https://github.com/apache/spark/blob/v2.4.2/dev/make-distribution.sh

1. 环境

• java 1.8
• maven 3.6.1
• scala 2.12.8

2. 下载源码包

• spark2.4.2
  链接:spark-2.4.2.tgz
  解压 tar -zxvf spark-2.4.2.tgz
  权限 chown -R hadoop:hadoop spark-2.4.2
• scala2.12.8
  链接:scala-2.12.8.tgz
  解压 tar -zxvf scala-2.12.8.tgz
  权限 chown -R root:root scala-2.12.8
• maven3.6.1
  链接:maven3.6.1
  tar -zxvf apache-maven-3.6.1-bin.tar.gz
  权限 chown -R hadoop:hadoop apache-maven-3.6.1

3. 环境变量

• vi .bash_profile

export JAVA_HOME=/usr/java/jdk1.8.0_45
#export JAVA_HOME=/usr/java/jdk1.7.0_45
export SCALA_HOME=/usr/scala/scala-2.12.8
#export MAVEN_HOME=/usr/maven/apache-maven-3.3.9
export MAVEN_HOME=/usr/maven/apache-maven-3.6.1
export MAVEN_OPTS="-Xms1024m -Xmx2048m"
export PROTOBUF_HOME=/usr/protobuf
export FINDBUGS_HOME=/home/hadoop/lib/findbugs-1.3.9
export HADOOP_HOME=/home/hadoop/app/hadoop-2.6.0-cdh5.7.0
export HIVE_HOME=/home/hadoop/app/hive-1.1.0-cdh5.7.0
export MAVEN_OPTS="-Xmx2g -XX:ReservedCodeCacheSize=512m"


export PATH=$SCALA_HOME/bin:$HIVE_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$FINDBUGS_HOME/bin:$PROTOBUF_HOME/bin:$MAVEN_HOME/bin:$JAVA_HOME/bin:$PATH

• source .bash_profile

4. maven 配置

参考博客maven配置参数
5. ## 检测环境

[hadoop@hadoop001 source]$ scala -version
Scala code runner version 2.12.8 -- Copyright 2002-2018, LAMP/EPFL and Lightbend, Inc.
[hadoop@hadoop001 spark-2.4.2]$ mvn -version
Apache Maven 3.6.1 (d66c9c0b3152b2e69ee9bac180bb8fcc8e6af555; 2019-04-04T19:00:29Z)
Maven home: /usr/maven/apache-maven-3.6.1

6. 修改pom文件

添加

<repository>
    <id>cloudera</id>
    <url>https://repository.cloudera.com/artifactory/cloudera-repos/</url>
</repository>

7. 开始编译

[hadoop@hadoop001 spark-2.4.2]$ ./dev/make-distribution.sh --name 2.6.0-cdh5.7.0  --tgz  -Phadoop-2.6 -Phive -Phive-thriftserver  -Pyarn -Pkubernetes -Dhadoop.version=2.6.0-cdh5.7.0

等待一段时间我是大约半小时
出现一下信息说明编译成功

查看文件

• ok编译完成