yarn jobhistory 服务
mapred-site.xml
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<property>
<name>mapreduce.application.classpath</name>
<value>$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*</value>
</property>
<property>
<name>mapreduce.jobhistory.address</name>
<value>mustafa-PC:10020</value>
</property>
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>mustafa-PC:19888</value>
</property>
</configuration>
yarn-site.xml
<configuration>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.env-whitelist</name>
<value>JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOME</value>
</property>
<property>
<name>yarn.resourcemanager.hostname</name>
<value>mustafa-PC</value>
</property>
<property>
<name>yarn.resourcemanager.hostname</name>
<value>mustafa-PC</value>
</property>
<property>
<name>yarn.log-aggregation-enable</name>
<value>true</value>
</property>
<property>
<name>yarn.log-aggregation.retain-seconds</name>
<value>86400</value>
</property>
<property>
<name>yarn.log.server.url</name>
<value>http://mustafa-PC:19888/jobhistory/job/</value>
</property>
</configuration>
hadoop开始jobhistory服务器运行命令
sbin/mr-jobhistory-daemon.sh start historyserver
history server 地址 http://mustafa-pc:19888/jobhistory
spark jobhistory服务
创建spark 日志存储位置
bin/hdfs dfs -mkdir -p /spark/history
spark-env.sh
SPARK_HISTORY_OPTS="-Dspark.history.fs.logDirectory=hdfs://mustafa-PC:9000/spark/history -Dspark.history.ui.port=18080 -Dspark.history.fs.cleaner.enabled=true -Dspark.history.fs.cleaner.interval=1d -Dspark.history.fs.cleaner.maxAge=7d"
启动history server服务
sbin/start-history-server.sh
history 服务地址:http://mustafa-pc:18080/
spark-default.conf文件
spark.master yarn
spark.eventLog.enabled true
spark.eventLog.dir hdfs://mustafa-PC:9000/spark/history
yarn history server 整合 spark history server
spark-default.conf文件
spark.yarn.historyServer.address http://mustafa-PC:18080