spark任务提交报错

最新推荐文章于 2024-04-23 21:30:55 发布

WQQ72600

最新推荐文章于 2024-04-23 21:30:55 发布

阅读量2.1k

点赞数 1

分类专栏： spark

本文链接：https://blog.csdn.net/WQQ72600/article/details/90376465

版权

本文讨论了Spark任务提交时遇到的问题，重点分析了YARN配置中的`yarn.scheduler.maximum-allocation-mb`和`yarn.nodemanager.resource.memory-mb`参数。`yarn.scheduler.maximum-allocation-mb`设定容器最大内存限制，超过此值会导致资源请求异常；而`yarn.nodemanager.resource.memory-mb`定义了节点可供分配的物理内存总量，默认为8192MB。正确配置这些参数有助于避免提交失败并优化集群资源使用。

摘要由CSDN通过智能技术生成

Exception in thread "main" java.lang.IllegalArgumentException: Required executor memory (8192+819 MB) is above the max threshold (8192 MB) of this cluster! Please check the values of 'yarn.scheduler.maximum-allocation-mb' and/or 'yarn.nodemanager.resource.memory-mb'.
	at org.apache.spark.deploy.yarn.Client.verifyClusterResources(Client.scala:319)
	at org.apache.spark.deploy.yarn.Client.submitApplication(Client.scala:167)
	at org.apache.spark.deploy.yarn.Client.run(Client.scala:1109)
	at org.apache.spark.deploy.yarn.Client$.main(Client.scala:1168)
	at org.apache.spark.deploy.yarn.Client.main(Client.scala)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$de