spark的内存过小报错

提交任务

spark-submit --class org.apache.spark.examples.SparkPi  --master yarn  --deploy-mode client  --driver-memory 600M  --executor-memory 500M  --num-executors 1  /usr/local/spark/examples/jars/spark-examples_2.11-2.3.0.jar  3

console报错

2020-08-11 10:39:50 INFO  YarnSchedulerBackend$YarnSchedulerEndpoint:54 - ApplicationMaster registered as NettyRpcEndpointRef(spark-client://YarnAM)
2020-08-11 10:39:53 WARN  YarnSchedulerBackend$YarnSchedulerEndpoint:66 - Requesting driver to remove executor 1 for reason Container marked as failed: container_1597065725323_0004_02_000002 on host: bj3-dev-search-02.tencn. Exit status: 1. Diagnostics: Exception from container-launch.
Container id: container_1597065725323_0004_02_000002
Exit code: 1
Stack trace: ExitCodeException exitCode=1:
	at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
	at org.apache.hadoop.util.Shell.run(Shell.java:455)
	at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:715)
	at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:211)
	at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
	at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)


Container exited with a non-zero exit code 1

查看nodemanager的yarn日志

java.lang.IllegalArgumentException: System memory 466092032 must be at least 471859200. Please increase heap size using the --driver-memory option or spark.driver.memory in Spark configuration.
	at org.apache.spark.memory.UnifiedMemoryManager$.getMaxMemory(UnifiedMemoryManager.scala:217)
	at org.apache.spark.memory.UnifiedMemoryManager$.apply(UnifiedMemoryManager.scala:199)
	at org.apache.spark.SparkEnv$.create(SparkEnv.scala:330)
	at org.apache.spark.SparkEnv$.createDriverEnv(SparkEnv.scala:175)
	at org.apache.spark.SparkContext.createSparkEnv(SparkContext.scala:256)
	at org.apache.spark.SparkContext.<init>(SparkContext.scala:423)
	at org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:58)
	at com.kk.search.spark.SparkPi.main(SparkPi.java:27)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)

这个和container的内存分配有关系

https://www.jianshu.com/p/bb0bdcb26ccc

这个错误目前还是很难完全排查，使用的是yarn client模式
报异常的代码是在这里

 /**
   * Return the total amount of memory shared between execution and storage, in bytes.
   */
  private def getMaxMemory(conf: SparkConf): Long = {
    val systemMemory = conf.getLong("spark.testing.memory", Runtime.getRuntime.maxMemory)
    val reservedMemory = conf.getLong("spark.testing.reservedMemory",
      if (conf.contains("spark.testing")) 0 else RESERVED_SYSTEM_MEMORY_BYTES)
    val minSystemMemory = (reservedMemory * 1.5).ceil.toLong
    if (systemMemory < minSystemMemory) {
      throw new IllegalArgumentException(s"System memory $systemMemory must " +
        s"be at least $minSystemMemory. Please increase heap size using the --driver-memory " +
        s"option or spark.driver.memory in Spark configuration.")
    }

这里的systemMemory 实际上就是Runtime.getRuntime.maxMemory，理论上这个值应该等于(jvm设置的值减去一个survivor的值)。
在本地的idea中使用local模式运行的时候显示的值和预期的是一致的。但是在yarn上使用client模式则是不一致的,不太容易判断。
也就是当设置 --driver-memory 500M的时候对应的到了yarn中并不是这些，这是需要注意的。具体的实现暂时没有看到相关的分析，估计还有一些难度，毕竟不是java的，只能看一个大概。

在deploy-mode 为client的方式下，进行提交如下

[root@bj3-dev--03 search_jar]# spark-submit --class com.kk.search.spark.SparkPi  --master yarn  --deploy-mode client  --driver-memory 600M  --executor-memory 1500M  --conf spark.yarn.am.memory=1500M --num-executors 1  spark-1.0-SNAPSHOT.jar  30
# 控制台输出,做了一些修改，增加了一些shell日志和代码日志
----------------------will do submit
/usr/local/jdk1.8.0_91/bin/java -cp /usr/local/spark/conf/:/usr/local/spark/jars/*:/usr/local/hadoop/etc/hadoop/ -Xmx600M org.apache.spark.deploy.SparkSubmit --master yarn --deploy-mode client --conf spark.yarn.am.memory=1500M --conf spark.driver.memory=600M --class com.kk.search.spark.SparkPi --executor-memory 1500M --num-executors 1 spark-1.0-SNAPSHOT.jar 30
----------------------
2020-08-13 18:23:24 WARN  NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
# 这里的输出是driver端的main方法获取了一下当前的jvm的情况，可以看到，Runtime.getRuntime().maxMemory() 获取到的可用内存和-Xmx600M 并不相等，也不等于(Eden+Survivor*2+Old)
---------------------------
max: 559415296 (533.50 M)
Non-heap: -1 (-0.00 M)
Heap: 559415296 (533.50 M)
Pool: Code Cache (type Non-heap memory) = 251658240 (240.00 M)
Pool: Metaspace (type Non-heap memory) = -1 (-0.00 M)
Pool: Compressed Class Space (type Non-heap memory) = 1073741824 (1024.00 M)
Pool: PS Eden Space (type Heap memory) = 166723584 (159.00 M)
Pool: PS Survivor Space (type Heap memory) = 21495808 (20.50 M)
Pool: PS Old Gen (type Heap memory) = 419430400 (400.00 M)
---------------------------
***************

从上面看到我们设置了-Xmx600M之后，对应的driver通过Runtime.getRuntime().maxMemory()获取到的最大可用系统内存和jvm当前的最大内存并不一致。
也就是说Runtime.getRuntime().maxMemory()获取的并不是jvm的内存，jvm的内存有可能以某种方式进行了限制，Runtime.getRuntime().maxMemory()获取的内存是动态变化的，某种对jvm内存的使用方式可能导致可用内存减少。
具体是那种使用方式会导致减小还不太清楚，可能是用来做缓存之类的。

https://blog.csdn.net/u011564172/article/details/68496848
https://www.cnblogs.com/mengrennwpu/p/11754341.html
https://cloud.tencent.com/developer/article/1198464