Spark on Yarn模式下历史日志配置记录

Spark on Yarn运行的时候Job History Server的配置

1.配置如下：yarn-site.xml

<property>
<name>yarn.log-aggregation-enable</name>
<value>true</value>
</property>

<property>
<name>yarn.log.server.url</name>
<value>http://ZQ0:19888/jobhistory/logs</value>
</property>

2.重启yarn服务及mr job history服务

3. 配置spark的conf/spark-defaults.conf文件内容

vim conf/spark-defaults.conf
spark.yarn.historyServer.address      
http://ZQ0:18080

4. 重新运行spark on yarn应用即可

./bin/spark-submit
–class com.ZQ.bigdata.spark.app.core.PVAndUVSparkCore
–master yarn
–deploy-mode cluster
–conf spark.yarn.submit.waitAppCompletion=false
/home/ZQ0/logs-analyzer-1.0.jar

备注：在提交任务之前如下进程需要启动好。

Namenode

DateNode

HistoryServer

job HistoryServer

ResourceManager

NodeManager

 

寄语专区：

转载请注明出处。