控制台显示application占用大，Sparkthriftserver启用及优化

./start-thriftserver.sh --executor-memory 6g --total-executor-cores 6

查看控制台发现 没有资源,修改 spark-defaults.conf

spark.deploy.defaultCores 2

https://www.cnblogs.com/cuishuai/p/7693417.html

Dynamic allocation of executors requires the external shuffle service. You may enable this through spark.shuffle.service.enabled.
https://www.jianshu.com/p/5749ad5d48e6

spark.dynamicAllocation.enabled true
spark.dynamicAllocation.maxExecutors 6
spark.dynamicAllocation.minExecutors 1

博主公众号
求关注