yarn/spark 历史服务器 jobhistory的启动以及整合

yarn jobhistory 服务

mapred-site.xml

<configuration>
    <property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
    </property>
    <property>
        <name>mapreduce.application.classpath</name>
        <value>$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*</value>
    </property>
    <property>
        <name>mapreduce.jobhistory.address</name>
        <value>mustafa-PC:10020</value>
    </property>
    <property>
        <name>mapreduce.jobhistory.webapp.address</name>
        <value>mustafa-PC:19888</value>
    </property>
</configuration>

yarn-site.xml

<configuration>
    <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>
    <property>
        <name>yarn.nodemanager.env-whitelist</name>
        <value>JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOME</value>
    </property>
    <property>
        <name>yarn.resourcemanager.hostname</name>
        <value>mustafa-PC</value>
    </property>
    <property>
        <name>yarn.resourcemanager.hostname</name>
        <value>mustafa-PC</value>
    </property>
    <property>
        <name>yarn.log-aggregation-enable</name>
        <value>true</value>
    </property>
    <property>
        <name>yarn.log-aggregation.retain-seconds</name>
        <value>86400</value>
    </property>
    <property>
        <name>yarn.log.server.url</name>
        <value>http://mustafa-PC:19888/jobhistory/job/</value>
    </property>
</configuration>

hadoop开始jobhistory服务器运行命令

sbin/mr-jobhistory-daemon.sh start historyserver 

history server 地址 http://mustafa-pc:19888/jobhistory 

spark jobhistory服务

创建spark 日志存储位置

bin/hdfs dfs -mkdir -p /spark/history

spark-env.sh

SPARK_HISTORY_OPTS="-Dspark.history.fs.logDirectory=hdfs://mustafa-PC:9000/spark/history -Dspark.history.ui.port=18080 -Dspark.history.fs.cleaner.enabled=true -Dspark.history.fs.cleaner.interval=1d -Dspark.history.fs.cleaner.maxAge=7d"

启动history server服务

sbin/start-history-server.sh

history 服务地址：http://mustafa-pc:18080/

spark-default.conf文件

spark.master                     yarn
spark.eventLog.enabled           true
spark.eventLog.dir               hdfs://mustafa-PC:9000/spark/history

yarn history server 整合 spark history server

spark-default.conf文件

spark.yarn.historyServer.address    http://mustafa-PC:18080