Flink集群之Current usage: 106.2 MB of 1 GB physical memory used; 2.1 GB of 2.1 GB virtual memory used

出错问题报告

org.apache.flink.client.deployment.ClusterDeploymentException: Couldn't deploy Yarn session cluster
	at org.apache.flink.yarn.AbstractYarnClusterDescriptor.deploySessionCluster(AbstractYarnClusterDescriptor.java:423)
	at org.apache.flink.yarn.cli.FlinkYarnSessionCli.run(FlinkYarnSessionCli.java:607)
	at org.apache.flink.yarn.cli.FlinkYarnSessionCli.lambda$main$2(FlinkYarnSessionCli.java:810)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1692)
	at org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
	at org.apache.flink.yarn.cli.FlinkYarnSessionCli.main(FlinkYarnSessionCli.java:810)
Caused by: org.apache.flink.yarn.AbstractYarnClusterDescriptor$YarnDeploymentException: The YARN application unexpectedly switched to state FAILED during deployment. 
Diagnostics from YARN: Application application_1585641509397_0006 failed 1 times due to AM Container for appattempt_1585641509397_0006_000001 exited with  exitCode: -103
For more detailed output, check application tracking page:http://master:8088/proxy/application_1585641509397_0006/Then, click on links to logs of each attempt.
Diagnostics: Container [pid=1925,containerID=container_1585641509397_0006_01_000001] is running beyond virtual memory limits. Current usage: 106.2 MB of 1 GB physical memory used; 2.1 GB of 2.1 GB virtual memory used. Killing container.
Dump of the process-tree for container_1585641509397_0006_01_000001 :
	|- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
	|- 1939 1925 1925 1925 (java) 269 223 2187988992 26862 /app/jdk1.8.0_191/bin/java -Xmx424m -Dlog.file=/app/hadoop-2.6.4/logs/userlogs/application_1585641509397_0006/container_1585641509397_0006_01_000001/jobmanager.log -Dlogback.configurationFile=file:logback.xml -Dlog4j.configuration=file:log4j.properties org.apache.flink.yarn.entrypoint.YarnSessionClusterEntrypoint 
	|- 1925 1923 1925 1925 (bash) 0 0 108609536 332 /bin/bash -c /app/jdk1.8.0_191/bin/java -Xmx424m  -Dlog.file=/app/hadoop-2.6.4/logs/userlogs/application_1585641509397_0006/container_1585641509397_0006_01_000001/jobmanager.log -Dlogback.configurationFile=file:logback.xml -Dlog4j.configuration=file:log4j.properties org.apache.flink.yarn.entrypoint.YarnSessionClusterEntrypoint  1> /app/hadoop-2.6.4/logs/userlogs/application_1585641509397_0006/container_1585641509397_0006_01_000001/jobmanager.out 2> /app/hadoop-2.6.4/logs/userlogs/application_1585641509397_0006/container_1585641509397_0006_01_000001/jobmanager.err 

Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Failing this attempt. Failing the application.
If log aggregation is enabled on your cluster, use this command to further investigate the issue:
yarn logs -applicationId application_1585641509397_0006
	at org.apache.flink.yarn.AbstractYarnClusterDescriptor.startAppMaster(AbstractYarnClusterDescriptor.java:1065)
	at org.apache.flink.yarn.AbstractYarnClusterDescriptor.deployInternal(AbstractYarnClusterDescriptor.java:545)
	at org.apache.flink.yarn.AbstractYarnClusterDescriptor.deploySessionCluster(AbstractYarnClusterDescriptor.java:416)
	... 7 more

------------------------------------------------------------
 The program finished with the following exception:

org.apache.flink.client.deployment.ClusterDeploymentException: Couldn't deploy Yarn session cluster
	at org.apache.flink.yarn.AbstractYarnClusterDescriptor.deploySessionCluster(AbstractYarnClusterDescriptor.java:423)
	at org.apache.flink.yarn.cli.FlinkYarnSessionCli.run(FlinkYarnSessionCli.java:607)
	at org.apache.flink.yarn.cli.FlinkYarnSessionCli.lambda$main$2(FlinkYarnSessionCli.java:810)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1692)
	at org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
	at org.apache.flink.yarn.cli.FlinkYarnSessionCli.main(FlinkYarnSessionCli.java:810)
Caused by: org.apache.flink.yarn.AbstractYarnClusterDescriptor$YarnDeploymentException: The YARN application unexpectedly switched to state FAILED during deployment. 
Diagnostics from YARN: Application application_1585641509397_0006 failed 1 times due to AM Container for appattempt_1585641509397_0006_000001 exited with  exitCode: -103
For more detailed output, check application tracking page:http://master:8088/proxy/application_1585641509397_0006/Then, click on links to logs of each attempt.
Diagnostics: Container [pid=1925,containerID=container_1585641509397_0006_01_000001] is running beyond virtual memory limits. Current usage: 106.2 MB of 1 GB physical memory used; 2.1 GB of 2.1 GB virtual memory used. Killing container.
Dump of the process-tree for container_1585641509397_0006_01_000001 :
	|- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
	|- 1939 1925 1925 1925 (java) 269 223 2187988992 26862 /app/jdk1.8.0_191/bin/java -Xmx424m -Dlog.file=/app/hadoop-2.6.4/logs/userlogs/application_1585641509397_0006/container_1585641509397_0006_01_000001/jobmanager.log -Dlogback.configurationFile=file:logback.xml -Dlog4j.configuration=file:log4j.properties org.apache.flink.yarn.entrypoint.YarnSessionClusterEntrypoint 
	|- 1925 1923 1925 1925 (bash) 0 0 108609536 332 /bin/bash -c /app/jdk1.8.0_191/bin/java -Xmx424m  -Dlog.file=/app/hadoop-2.6.4/logs/userlogs/application_1585641509397_0006/container_1585641509397_0006_01_000001/jobmanager.log -Dlogback.configurationFile=file:logback.xml -Dlog4j.configuration=file:log4j.properties org.apache.flink.yarn.entrypoint.YarnSessionClusterEntrypoint  1> /app/hadoop-2.6.4/logs/userlogs/application_1585641509397_0006/container_1585641509397_0006_01_000001/jobmanager.out 2> /app/hadoop-2.6.4/logs/userlogs/application_1585641509397_0006/container_1585641509397_0006_01_000001/jobmanager.err 

Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Failing this attempt. Failing the application.
If log aggregation is enabled on your cluster, use this command to further investigate the issue:
yarn logs -applicationId application_1585641509397_0006
	at org.apache.flink.yarn.AbstractYarnClusterDescriptor.startAppMaster(AbstractYarnClusterDescriptor.java:1065)
	at org.apache.flink.yarn.AbstractYarnClusterDescriptor.deployInternal(AbstractYarnClusterDescriptor.java:545)
	at org.apache.flink.yarn.AbstractYarnClusterDescriptor.deploySessionCluster(AbstractYarnClusterDescriptor.java:416)
	... 7 more
2020-03-31 15:17:07,311 INFO  org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Cancelling deployment from Deployment Failure Hook
2020-03-31 15:17:07,311 INFO  org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Killing YARN application
2020-03-31 15:17:07,324 INFO  org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Deleting files in hdfs://master:9000/user/root/.flink/application_1585641509397_0006.

出错点

遇到问题不要慌，仔细看报错，“Diagnostics:…” 错误很明显的
原因是Current usage: 106.2 MB of 1 GB physical memory used; 2.1 GB of 2.1 GB virtual memory used. Killing container
字面原因是容器内存不够，实际上是flink on yarn启动时检查虚拟内存造成的
解决方案：
修改配置文件，让它不检查就没事了
修改etc/hadoop/yarn-site.xml

<property>
	<name>yarn.nodemanager.vmem-check-enabled</name>
    <value>false</value>
</property>