SPARK 2.2.1 YARN安装部署

1、安装scala：

在SparkNode01~03节点安装：

cd /opt

tar xzvf scala-2.11.12.tgz

mv scala-2.11.12 scala

scp -r scala @SparkNode03:/opt

vi /etc/profile

SCALA_HOME=/opt/scala

PATH=$PATH:$SCALA_HOME/bin

source /etc/profile 

scala -version

2、安装Spark

在SparkNode01~03节点安装：

cd /opt

tar xzvf spark-2.2.1-bin-hadoop2.7.tgz

mv spark-2.2.1-bin-hadoop2.7 spark

cd spark/conf

cp spark-env.sh.template spark-env.sh

vi spark-env.sh

添加：

export JAVA_HOME=/usr/local/java/jdk1.8.0_121

export SCALA_HOME=/opt/scala

export HADOOP_HOME=/opt/hadoop

export HADOOP_CONF_DIR=/opt/hadoop/tc/hadoop

export SPARK_MASTER_PORT=7077

export SPARK_EXECUTOR_CORES=2

export SPARK_EXECUTOR_MEMORY=1024m

export SPARK_EXECUTOR_INSTANCES=1

 

cp slaves.template slaves

vi slaves

添加：

SparkNode01

SparkNode02

SparkNode03

 

在hdfs上传spark的jar包，并修改/opt/spark/conf/spark-defaults.conf

cd /opt/hadoop/bin

./hadoop fs -mkdir /spark_jars

./hadoop fs -put  /opt/spark/jars/* /spark_jars

cp spark-defaults.conf.template spark-defaults.conf

vi spark-defaults.conf

添加：

spark.yarn.jars=hdfs://Master01:9000/spark_jars/*

 

验证：

./spark-submit --class org.apache.spark.examples.SparkPi --master yarn --deploy-mode cluster --driver-memory 1g --executor-memory 1g --executor-cores 1 --num-executors 3 /opt/spark/examples/jars/spark-examples_2.11-2.2.1.jar 10

转载于:https://www.cnblogs.com/weidafei/p/8461758.html