spark之yarn模式搭建
1.安装
先到官网下载spark:https://mirrors.bfsu.edu.cn/apache/
解压
tar -zxvf spark-3.0.0-bin-hadoop2.7.tgz -C /opt/software
2.配置
2.1 conf的spark-env.sh
cd spark/conf
cp spark-env.sh.template spark-env.sh
vi spark-env.sh
#添加以下内容
export YARN_CONF_DIR=/opt/links/hadoop/etc/hadoop
2.2 修改环境变量
####__SPARK_HOME___###
export SPARK_HOME=/opt/software/spark-yarn
export PARH=${PATH}:${SPARK_HOME}/bin
3. 使用
命令提交
bin/spark-submit \
--class org.apache.spark.examples.SparkPi \
--master yarn \
--deploy-mode cluster \
./examples/jars/spark-examples_2.12-3.0.0.jar \
10
spark-shell
cd spark
bin/spark-shell