Spark jobhistory日志清理

spark开启history-server之后，app日志会保存在制定的目录下，
若yarn开启日志收集，也需要进行app-logs的自动清理,因为项目组大数据平台经常日志太多，需要做清理，jobhistory中 也会经常出现长时间的load，鉴于此问题采取以下措施解决

## yarn日志收集

yarn.log-aggregation.retain-seconds = 1209600

yarn.log-aggregation.retain-check-interval-seconds = 86400





## spark-history日志

spark.history.fs.cleaner.enabled = true

spark.history.fs.cleaner.interval = 1d

spark.history.fs.cleaner.maxAge = 14d



## spark2-history日志

spark.history.fs.cleaner.enabled = true

spark.history.fs.cleaner.interval = 1d

spark.history.fs.cleaner.maxAge = 14d

转载出处：

https://blog.csdn.net/u011098327/article/details/81077370