Required executor memory (1024+384 MB) is above the max threshold (1024 MB) of this cluster!

最新推荐文章于 2023-10-28 20:41:49 发布

weixin_33801856

最新推荐文章于 2023-10-28 20:41:49 发布

阅读量2.4k

点赞数

文章标签： scala 大数据 java

原文链接：https://my.oschina.net/lwenhao/blog/3022853

版权

博客内容提到了在运行时遇到'Required executor memory (1024+384 MB) is above the max threshold (1024 MB) of this cluster!'的错误。解决方法是将executor内存和driver内存的默认值调整为2GB，并重启集群。

摘要由CSDN通过智能技术生成

2019独角兽企业重金招聘Python工程师标准>>>

Required executor memory (1024+384 MB) is above the max threshold (1024 MB) of this cluster!

问题：

运行spark-shell时报错：

[root[@master88](https://my.oschina.net/u/2691764) ~]# spark-shell 
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel).
Welcome to
	  ____              __
	 / __/__  ___ _____/ /__
	_\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 1.6.0
	  /_/

Using Scala version 2.10.5 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_201)
Type in expressions to have them evaluated.
Type :help for more information.
19/03/15 10:13:12 ERROR spark.SparkContext: Error initializing SparkContext.
java.lang.IllegalArgumentException: Required executor memory (1024+384 MB) is above the max threshold (1024 MB) of this cluster! Please check the values of 'yarn.scheduler.maximum-allocation-mb' and/or 'yarn.nodemanager.resource.memory-mb'.
	at org.apache.spark.deploy.yarn.Client.verifyClusterResources(Client.scala:281)
	at org.apache.spark.deploy.yarn.Client.submitApplication(Client.scala:140)
	at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:57)
	at org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:151)
	at org.apache.spark.SparkContext.<init>(SparkContext.scala:538)
	at org.apache.spark.repl.SparkILoop.createSparkContext(SparkILoop.scala:1022)
	at $line3.$read$$iwC$$iwC.<init>(<console>:15)
	at $line3.$read$$iwC.<init>(<console>:25)
	at $line3.$read.<init>(<console>:27)
	at $line3.$read$.<init>(<console>:31)
	at $line3.$read$.<clinit>(<console>)
	at $line3.$eval$.<init>(<console>:7)
	at $line3.$eval$.<clinit>(<console>)
	at $line3.$eval.$print(<console>)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:1045)
	at org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1326)
	at org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:821)
	at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:852)
	at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:800)
	at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:857)
	at org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:902)
	at org.apache.spark.repl.SparkILoop.command(SparkILoop.scala:814)
	at org.apache.spark.repl.SparkILoopInit$$anonfun$initializeSpark$1.apply(SparkILoopInit.scala:125)
	at org.apache.spark.repl.SparkILoopInit$$anonfun$initializeSpark$1.apply(SparkILoopInit.scala:124)
	at org.apache.spark.repl.SparkIMain.beQuietDuring(SparkIMain.scala:305)
	at org.apache.spark.repl.SparkILoopInit$class.initializeSpark(SparkILoopInit.scala:124)
	at org.apache.spark.repl.SparkILoop.initializeSpark(SparkILoop.scala:64)
	at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1$$anonfun$apply$mcZ$sp$5.apply$mcV$sp(SparkILoop.scala:974)
	at org.apache.spark.repl.Spar