Spark Job记录查看

修改配置文件

cd /opt/spark/conf
cp spark-defaults.conf.template spark-defaults.conf

vi spark-defaults.conf

#开启日志服务
spark.eventLog.enabled true 
#日志存放位置
spark.eventLog.dir hdfs://master:8020/var/log/spark 
#日志可压缩
spark.eventLog.compress true

vi spark-env.sh

export SPARK_HISTORY_OPTS=”-Dspark.history.ui.port=18080 -Dspark.history.retainedApplications=3 -Dspark.history.fs.logDirectory=hdfs:/master:8020/var/log/spark”

进入/opt/spark/sbin,启动服务

./stop-history-server.sh

提交作业进行测试

bin/spark-submit \
--class org.apache.spark.examples.SparkPi \
--master spark://192.168.100.10:7077 \
--executor-memory 1G \
--total-executor-cores 2 \
/opt/spark/examples/jars/spark-examples_2.11-2.4.3.jar \
100

访问ip:18080,查看作业历史纪录

192.168.100.10:18080