Spark编译

1、下载组件

1.1下载解压maven

	http://mirrors.shu.edu.cn/apache/maven/maven-3/3.3.9/binaries/

1.1.2 修改setting.xml
修改jar包保存目录

<localRepository>/home/hadoop/.m2/repository</localRepository>

添加镜像

 <repository>
	<id>central-repos</id>
	<name>Central Repository</name>
	<url>http://repo.maven.apache.org/maven2</url>
</repository>
<repository>
	<id>alimaven</id>
	<name>aliyun maven</name>
	<url>http://maven.aliyun.com/nexus/content/groups/public/</url>          
</repository>

1.2、下载解压JDK

Oracle官网下载1.7+版本

1.3、下载解压Scala

https://www.scala-lang.org/download/2.11.8.html

1.4、下载解压Spark包

https://archive.apache.org/dist/spark/spark-2.1.3/

2、配置环境变量

vim ~/.bash_profile

export PATH
export JAVA_HOME=/opt/modules/jdk1.8.0_171
export PATH=$PATH:$JAVA_HOME/bin:$JAVA_HOME/jre/bin
export CLASSPATH=.:$JAVA_HOME/lib:$JAVA_HOME/jre/lib:$CLASSPATH

export MAVEN_HOME=/opt/modules/apache-maven-3.3.9
export PATH=$PATH:$MAVEN_HOME/bin

export HADOOP_HOME=/opt/modules/hadoop-2.6.0-cdh5.13.1
export HADOOP_INSTALL=$HADOOP_HOME
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export HADOOP_COMMON_HOME=$HADOOP_HOME
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
export HADOOP_HDFS_HOME=$HADOOP_HOME
export HADOOP_YARN_HOME=$HADOOP_HOME
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin

export SCALA_HOME=/opt/modules/scala-2.11.8
export PATH=$PATH:$SCALA_HOME/bin

source ~/.bash_profile

3、修改make-distribution.sh

将变量修改为常量，提高编译效率

vim make-distribution.sh

VERSION=2.1.3
SCALA_VERSION=2.11   #必须为大版本
SPARK_HADOOP_VERSION=2.6.0-cdh5.13.1
SPARK_HIVE=1   #表示支持hive,0不支持

4、修改pom.xml

4、编译

将scala包复制至spark2.1.3/build目录，然后执行编译命令

./dev/make-distribution.sh --name 2.6.0-cdh5.13.1 --tgz -Phadoop-2.6 -Dhadoop.version=2.6.0-cdh5.13.1 -Phive -Phive-thriftserver -Pmesos -Pyarn

-Phive 表示支持hive
-Pyarn 表示支持yarn