spark-submit报错

spark-submit --class  ${main_class} \
--master yarn \
--deploy-mode client \
--driver-memory 8g \
--executor-memory 6g \
--num-executors 1 \
--executor-cores 34 \
 ${HOME_PATH}/obs_finance-1.0-SNAPSHOT-jar-with-dependencies.jar ${year_month}

注意这个34超过了我们yarn设置的最大32

 不出意外就报错了

Caused by: java.io.IOException: Failed to send RPC RPC 6675935870406582041 to /9.135.247.xxx:36633: io.netty.channel.StacklessClosedChannelException


23/03/07 10:22:22 ERROR cluster.YarnSchedulerBackend$YarnSchedulerEndpoint: Sending RequestExecutors(1,0,Map(),Set()) to AM was unsuccessful


23/03/07 10:22:22 ERROR util.Utils: Uncaught exception in thread main
org.apache.spark.SparkException: Exception thrown in awaitResult: 

Caused by: java.io.IOException: Connection reset by peer

我们分析报错

rpc / Connection reset by peer/网络原因？其实不是。

 Sending RequestExecutors(1,0,Map(),Set()) to AM was unsuccessful 这句话才是最主要的。

就是向AM申请executors资源的时候失败了，为啥失败，网络原因？还是资源超过上限，按照我们瞎搞的肯定就是资源不够了。

所以合理分配资源。

------------------------------------------------------------------------------------------------------------------------