Spark整合Yarn历史服务器

最新推荐文章于 2024-07-20 22:06:32 发布

shenjies88

最新推荐文章于 2024-07-20 22:06:32 发布

阅读量261

点赞数

分类专栏：运维文章标签： spark 服务器大数据

本文链接：https://blog.csdn.net/qq_37461154/article/details/128959802

版权

运维专栏收录该内容

1 篇文章 0 订阅

订阅专栏

yarn-site.xml

    <property>
        <name>yarn.log-aggregation-enable</name>
        <value>true</value>
    </property>
    <property>
        <name>yarn.log-aggregation.retain-seconds</name>
        <value>604800</value>
    </property>
    <property>
        <name>yarn.nodemanager.pmem-check-enabled</name>
        <value>false</value>
    </property>
    <property>
        <name>yarn.nodemanager.vmem-check-enabled</name>
        <value>false</value>
    </property>

spark-env.sh

JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.292.b10-1.el7_9.x86_64
HADOOP_CONF_DIR=/root/software/hadoop/etc/hadoop
YARN_CONF_DIR=/root/software/hadoop/etc/hadoop

SPARK_HISTORY_OPTS="-Dspark.history.fs.logDirectory=hdfs://hadoop110:9000/sparklog/ -Dspark.history.fs.cleaner.enabled=true"

spark-defaults.conf

spark.eventLog.enabled  true
spark.eventLog.dir      hdfs://hadoop110:9000/sparklog/
spark.eventLog.compress true
spark.yarn.historyServer.address hadoop110:18080
spark.yarn.jars hdfs://hadoop110:9000/spark/jars/*

创建HDFS目录

/sparklog，/spark/jars

提交Jar包

bin/hadoop fs -put /root/software/spark/jars/* /spark/jars/

启动服务

mr-jobhistory-daemon.sh start historyserver

sbin/start-history-server.sh

测试命令

bin/spark-submit --master yarn --deploy-mode cluster --driver-memory 512m --executor-memory 512m --num-executors 1 --class org.apache.spark.examples.SparkPi examples/jars/spark-examples_2.12-3.3.1.jar 10