由于spark1.1.0还没正式发布只是v1.1.0-snapshot2 版本。但是hadoop2.4.1的resourcemanager ha 在spark 1.0.2版本还不支持(提交spark到yarn的时候。applicationmaster老是会请求本地的8030端口)。有兴趣的同志可以先提现体验一下。。
注意:spark 得在集群机器都安装哦。路径都一样。
1:进入解压后的spark源码目录
cd /home/maven/spark-1.1.0-snapshot2
因为编译spark 需要很大内存。下载依赖也很花时间,而且很多依赖下载过程就卡死了。。,得一直取消,再重复编译。。。很可怕呢。。
export MAVEN_OPTS="-Xmx3g -XX:MaxPermSize=1024M -XX:ReservedCodeCacheSize=1024m"
修改 /home/maven/spark-1.1.0-snapshot2/pom.xml
找到 <protobuf.version>2.4.1</protobuf.version> 修改成你集群安装的protobuf版本 例如:我使用的是2.5.0<protobuf.version>2.5.0</protobuf.version>。不修改,编译完成后,提交job会找不到hadoop 的native等错误。。
2:进行编译
./make-distribution.sh --tgz --skip-java-test
-Dyarn.version=2.4.1
-Dhadoop.version
=
2.4.1
-Pyarn
3:编译成成功后生成的文件如下:
/home/maven/spark-1.1.0-snapshot2/spark-1.1.0-bin-.tgz
4:将编译好后的spark 文件copy到spark的home目录
mkdir /opt/spark
mkdir /opt/spark/lib
/home/maven/spark-1.1.0-snapshot2/spark-1.1.0-bin-.tgz /opt/spark
tar -zxvf /opt/spark/spark-1.1.0-bin-.tgz
mv /home/maven/spark-1.1.0-bin-/* /opt/spark/
5创建spark用户,并将文件的权限赋予spark用户
useradd spark
password spark
chown -R spark.spark /opt/spark
6:为spark用户添加执行环境变量
export JAVA_HOME=/usr/java/jdk1.7.0_45
export PATH=$JAVA_HOME/bin:$PATH
export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
export HADOOP_INSTALL=/opt/hadoop
export HADOOP_HOME=$HADOOP_INSTALL
export PATH=$PATH:$HADOOP_INSTALL/bin
export PATH=$PATH:$HADOOP_INSTALL/sbin
export HADOOP_MAPRED_HOME=$HADOOP_INSTALL
export HADOOP_COMMON_HOME=$HADOOP_INSTALL
export HADOOP_HDFS_HOME=$HADOOP_INSTALL
export YARN_HOME=$HADOOP_INSTALL
export SCALA_HOME=/usr/local/scala
export PATH=$SCALA_HOME/bin:$PATH
export SPARK_HOME=/usr/local/spark
export PATH=$JAVA_HOME/bin:$PATH
export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
export HADOOP_INSTALL=/opt/hadoop
export HADOOP_HOME=$HADOOP_INSTALL
export PATH=$PATH:$HADOOP_INSTALL/bin
export PATH=$PATH:$HADOOP_INSTALL/sbin
export HADOOP_MAPRED_HOME=$HADOOP_INSTALL
export HADOOP_COMMON_HOME=$HADOOP_INSTALL
export HADOOP_HDFS_HOME=$HADOOP_INSTALL
export YARN_HOME=$HADOOP_INSTALL
export SCALA_HOME=/usr/local/scala
export PATH=$SCALA_HOME/bin:$PATH
export SPARK_HOME=/usr/local/spark
export SPARK_JAR=/usr/local/spark/lib/spark-assembly-1.0.2-hadoop2.4.1.jar
export PATH=$SPARK_HOME/bin:$PATH
7:修改conf目录下的spark-env.sh,添加环境变量
export HADOOP_CONF_DIR=$HADOOP_INSTALL/etc/hadoop
export YARN_CONF_DIR=$HADOOP_INSTALL/etc/hadoop
export SPARK_HOME=/opt/spark
export SPARK_JAR=/opt/spark/lib/spark-assembly-1.1.0-hadoop2.4.1.jar
export PATH=$SPARK_HOME/bin:$PATH
其中
export HADOOP_CONF_DIR=$HADOOP_INSTALL/etc/hadoop
export YARN_CONF_DIR=$HADOOP_INSTALL/etc/hadoop
export YARN_CONF_DIR=$HADOOP_INSTALL/etc/hadoop
这2个变量任选其一
SPARK_JAR 必须指定。
8提交job 的命令
./bin/spark-submit --class org.apache.spark.examples.SparkPi \
--master yarn-cluster \
--num-executors 1 \
--driver-memory 1g \
--executor-memory 1g \
--executor-cores 1 \
lib/spark-examples*.jar \
1
--master yarn-cluster \
--num-executors 1 \
--driver-memory 1g \
--executor-memory 1g \
--executor-cores 1 \
lib/spark-examples*.jar \
1