spark1.6编译

spark编译有3种

一.使用maven编译
mvn clean package \
-DskipTests-Phadoop-2.4 \
-Dhadoop.version=2.6.0 -Pyarn \
-Phive-0.13.1-Phive-thriftserver
二.使用 sbt编译（不做介绍）
三.打包编译make-distribution.sh
./make-distribution.sh –tgz\
-Phadoop-2.4-Dhadoop.version=2.60-cdh5.4.0\
-Pyarn\
-Phive-0.13.1-Phive-thriftserver

1.在maven3.3.9/conf vi setting.xml文件的
配置镜像

<mirror>
    <id>nexus-osc</id>
    <mirrorOf>*</mirrorOf>
    <name>Nexus osc</name>
    <url>http://maven.oschina.net/content/groups/public/</url>
</mirror>

2.配置/etc/resolve.conf 添加谷歌的域名解释器
namesever 8.8.8.8
nameserver 8.8.4.4

3.进入到spark源码下的make-distribution.sh将

VERSION=$("$MVN" help:evaluate -Dexpression=project.version $@ 2>/dev/null | grep -v "INFO" | tail -n 1)
SCALA_VERSION=$("$MVN" help:evaluate -Dexpression=scala.binary.version $@ 2>/dev/null\
    | grep -v "INFO"\
    | tail -n 1)
SPARK_HADOOP_VERSION=$("$MVN" help:evaluate -Dexpression=hadoop.version $@ 2>/dev/null\
    | grep -v "INFO"\
    | tail -n 1)
SPARK_HIVE=$("$MVN" help:evaluate -Dexpression=project.activeProfiles -pl sql/hive $@ 2>/dev/null\
    | grep -v "INFO"\
    | fgrep --count "<id>hive</id>";\
    # Reset exit status to 0, otherwise the script stops here if the last grep finds nothing\
    # because we use "set -o pipefail"
    echo -n)

这块代码注释掉
结果为

VERSION=1.6.3
SCALA_VERSION=2.10.6
SPARK_HADOOP_VERSION=2.6.0
SPARK_HIVE=1     #表示要不要hive 不是版本号
#VERSION=$("$MVN" help:evaluate -Dexpression=project.version $@ 2>/dev/null | grep -v "INFO" | tail -n 1)
#SCALA_VERSION=$("$MVN" help:evaluate -Dexpression=scala.binary.version $@ 2>/dev/null\
#    | grep -v "INFO"\
#    | tail -n 1)
#SPARK_HADOOP_VERSION=$("$MVN" help:evaluate -Dexpression=hadoop.version $@ 2>/dev/null\
#    | grep -v "INFO"\
#    | tail -n 1)
#SPARK_HIVE=$("$MVN" help:evaluate -Dexpression=project.activeProfiles -pl sql/hive $@ 2>/dev/null\
#    | grep -v "INFO"\
#    | fgrep --count "<id>hive</id>";\
#    # Reset exit status to 0, otherwise the script stops here if the last grep finds nothing\
#    # because we use "set -o pipefail"
#    echo -n)

4.执行./make-distribution.sh –tgz -Pyarn -Phadoop-2.4 -Dhadoop.version=2.6.0 -Phive-2.0.1 -Phive-thriftserver