bin/spark-submit
–class com.huawei.cluster
–master yarn-cluster
==–driver-cores 2 ==
==–driver-memory 30G ==
–conf spark.shuffle.service.ennabled=true
–conf spark.memory.storageFraction=0.30
–conf spark.memory.fraction=0.7
–conf spark.default.parallelism=2800
–conf spark.sql.shuffle.partitions1=1400
–conf spark.yarn.executor.memeoryOverhead=4096
==–executor-memory 30g \ ==
==–executor-cores 8 ==
==–num-executors 20 ==
/export/servers/spark-2.2.0-bin-2.6.0-cdh5.14.0/examples/jars/spark-examples_2.11-2.2.0.jar
100
问题1:这个任务一共需要多少的Cores和Memory,答案:162cores,630G(driver=30g),
问题2:程序运行时每个executor上的storage内存和execution内存多少,9g,21g
正确答案:6.3g 14.7g,请参考下面图示
一道 spark-submit 面试题
最新推荐文章于 2021-10-21 05:38:52 发布