Spark on Yarn 环境搭建

1. 搭建目的

将Spark程序运行在Yarn集群中, 由Yarn集群完成资源调度

2. Spark On Yarn环境所依赖组件及版本

spark-3.1.2
hadoop-3.3.0
apache-hive-3.1.2-bin
zookeeper-3.4.6
Python 3.8.8
三台Centos7机器: 主机名分别为：node1、node2、node3

3. 配置Spark运行环境

vim conf/spark-env.sh

export SPARK_MASTER_PORT=7077

SPARK_MASTER_WEBUI_PORT=8080
SPARK_WORKER_CORES=2
SPARK_WORKER_MEMORY=2g
SPARK_WORKER_PORT=7078
SPARK_WORKER_WEBUI_PORT=8081

SPARK_HISTORY_OPTS="-Dspark.history.fs.logDirectory=hdfs://node1:8020/sparklog/ 
-Dspark.history.fs.cleaner.enabled=true"
SPARK_DAEMON_JAVA_OPTS="-Dspark.deploy.recoveryMode=ZOOKEEPER -Dspark.deploy.zookeeper.url=node1:2181,node2:2181,node3:2181 -Dspark.deploy.zookeeper.dir=/spark-ha"


HADOOP_CONF_DIR=/export/server/hadoop/etc/hadoop/
YARN_CONF_DIR=/export/server/hadoop/etc/hadoop/

4.配置Yarn运行环境

vim hadoop/etc/hadoop/yarn-site.xml

<configuration>
    <!-- 指定YARN的主角色（ResourceManager）的地址 -->
    <property>
        <name>yarn.resourcemanager.hostname</name>
        <value>node1</value>
    </property>

    <!-- NodeManager上运行的附属服务。需配置成mapreduce_shuffle，才可运行MapReduce程序默认值："" -->
    <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>
    <!-- 设置yarn集群的内存分配方案 -->
    <property>
        <name>yarn.nodemanager.resource.memory-mb</name>
        <value>20480</value>
    </property>
    <property>
        <name>yarn.scheduler.minimum-allocation-mb</name>
        <value>2048</value>
    </property>
    <property>
        <name>yarn.nodemanager.vmem-pmem-ratio</name>
        <value>2.1</value>
    </property>

    <!-- 是否将对容器实施物理内存限制 -->
    <property>
        <name>yarn.nodemanager.pmem-check-enabled</name>
        <value>false</value>
    </property>

    <!-- 是否将对容器实施虚拟内存限制。 -->
    <property>
        <name>yarn.nodemanager.vmem-check-enabled</name>
        <value>false</value>
    </property>
    <!-- 开启日志聚集 -->
    <property>
        <name>yarn.log-aggregation-enable</name>
        <value>true</value>
    </property>

    <!-- 设置yarn历史服务器地址 -->
    <property>
        <name>yarn.log.server.url</name>
        <value>http://node1:19888/jobhistory/logs</value>
    </property>

    <!-- 保存的时间7天 -->
    <property>
        <name>yarn.log-aggregation.retain-seconds</name>
        <value>604800</value>
    </property>

    <!-- Site specific YARN configuration properties -->

5.Spark设置历史服务器地址

vim spark/conf/spark-defaults.conf
spark.eventLog.dir hdfs://node1:8020/sparklog/

spark.eventLog.enabled  true
spark.eventLog.dir      hdfs://node1:8020/sparklog/
spark.eventLog.compress true
spark.yarn.historyServer.address        node1:18080
spark.yarn.jars  hdfs://node1:8020/spark/jars/*

配置后, 需要在HDFS上创建 sparklog目录
hdfs dfs -mkdir -p /sparklog

6. 设置日志级别

vim spark/conf/log4j.properties
log4j.rootCategory=WARN, console

log4j.rootCategory=WARN, console
log4j.appender.console=org.apache.log4j.ConsoleAppender
log4j.appender.console.target=System.err
log4j.appender.console.layout=org.apache.log4j.PatternLayout
log4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n

# Set the default spark-shell log level to WARN. When running the spark-shell, the
# log level for this class is used to overwrite the root logger's log level, so that
# the user can have different defaults for the shell and regular Spark apps.
log4j.logger.org.apache.spark.repl.Main=WARN

# Settings to quiet third party logs that are too verbose
log4j.logger.org.sparkproject.jetty=WARN
log4j.logger.org.sparkproject.jetty.util.component.AbstractLifeCycle=ERROR
log4j.logger.org.apache.spark.repl.SparkIMain$exprTyper=INFO
log4j.logger.org.apache.spark.repl.SparkILoop$SparkILoopInterpreter=INFO
log4j.logger.org.apache.parquet=ERROR
log4j.logger.parquet=ERROR

7.配置Spark相关依赖包地址

当Spark Application应用提交运行在YARN上时，默认情况下，每次提交应用都需要将依赖Spark相关jar包上传到YARN 集群中，为了节省提交时间和存储空间，将Spark相关jar包上传到HDFS目录中，设置属性告知Spark Application应用。

hadoop fs -mkdir -p /spark/jars/
hadoop fs -put spark/jars/*.jar  /spark/jars/

vim spark/conf/spark-defaults.conf
添加:
spark.yarn.jars hdfs://node1:8020/spark/jars/*

spark.eventLog.enabled  true
spark.eventLog.dir      hdfs://node1:8020/sparklog/
spark.eventLog.compress true
spark.yarn.historyServer.address        node1:18080
spark.yarn.jars  hdfs://node1:8020/spark/jars/*

8.启动服务

启动服务：
HDFS、YARN、MRHistoryServer和Spark HistoryServer，
命令如下：

start-dfs.sh
start-yarn.sh
mr-jobhistory-daemon.sh start historyserver
spark/sbin/start-history-server.sh

Spark HistoryServer服务WEB UI页面地址：

9.提交测试

计算圆周率PI的Spark程序，提交到Yarn上运行，在Centos平台上直接用spark-sumbit脚本提交程序

SPARK_HOME=/export/server/spark
${SPARK_HOME}/bin/spark-submit \
--master yarn \
--conf "spark.pyspark.driver.python=/root/anaconda3/bin/python3" \
--conf "spark.pyspark.python=/root/anaconda3/bin/python3" \
${SPARK_HOME}/examples/src/main/python/pi.py \
10

设置资源信息，提交运行pi程序至YARN上

SPARK_HOME=/export/server/spark
${SPARK_HOME}/bin/spark-submit \
--master yarn \
--driver-memory 512m \
--executor-memory 512m \
--executor-cores 1 \
--num-executors 2 \
--queue default \
--conf "spark.pyspark.driver.python=/root/anaconda3/bin/python3" \
--conf "spark.pyspark.python=/root/anaconda3/bin/python3" \
${SPARK_HOME}/examples/src/main/python/pi.py \
10

10. 参考文章

Spark-Standalone高可用集群搭建