Spark安装与配置

• 安装Spark-2.4.0

  tar xf spark-2.4.0.tar.gz -C /env

将 spark-2.4.0/yarn 目录下的 spark-2.4.0-yarn-shuffle.jar 拷贝到/env/hadoop-2.9.2/share/hadoop/yarn/lib 目录下

递归更该spark目录的拥有者

chown -R warmfrog /env/spark-2.4.0 

• 配置spark 环境变量

  cd /env/spark-2.4.0/conf
  cp spark-env.sh.template spark-env.sh
  vim spark-env.sh

添加如下环境变量

export JAVA_HOME=/env/jdk1.8
export HADOOP_HOME=/env/hadoop-2.9.2
export HADOOP_CONF_DIR=/env/hadoop-2.9.2/etc/hadoop
export YARN_CONF_DIR=/env/hadoop-2.9.2/etc/hadoop
export SPARK_HOME=/env/spark-2.4.0
export SPARK_LIBRARY_PATH=.:$JAVA_HOME/lib:$JAVA_HOME/jre/lib:$HADOOP_HOME/lib/native
export SPARK_MASTER_IP=127.0.0.1
export SPARK_MASTER_HOST=hdfs
export SPARK_LOCAL_DIRS=/env/spark-2.4.0
export SPARK_YARN_USER_ENV&#