Flink 启动 yarn-session.sh

最新推荐文章于 2024-07-08 18:10:18 发布
最新推荐文章于 2024-07-08 18:10:18 发布
阅读量4.6k 收藏 1
点赞数
分类专栏： 环境搭建 文章标签： flink
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/qq_40471761/article/details/96610557
版权

1.内存不足

启动命令
bin/yarn-session.sh -n 2 -s 2 -jm 1024 -tm 1024

解决办法：将 hadoop 的检查虚拟内存关闭掉

在yarn-site.xml 中添加

<property>    
    <name>yarn.nodemanager.vmem-check-enabled</name>    
    <value>false</value>    
</property>  
错误信息
2019-07-20 17:18:25,512 INFO org.apache.hadoop.yarn.server.resourcemanager.RMAppManager$ApplicationSummary: appId=application_1563612848821_0005,name=test,user=hadoop,queue=default,state=FAILED,trackingUrl=http://mini1:8088/cluster/app/application_1563612848821_0005,appMasterHost=N/A,startTime=1563614297058,finishTime=1563614305505,finalStatus=FAILED
2019-07-20 17:25:12,967 INFO org.apache.hadoop.yarn.server.resourcemanager.ClientRMService: Allocated new applicationId: 6
2019-07-20 17:25:16,952 INFO org.apache.hadoop.yarn.server.resourcemanager.ClientRMService: Application with id 6 submitted by user hadoop
2019-07-20 17:25:16,952 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: Storing application with id application_1563612848821_0006
2019-07-20 17:25:16,952 INFO org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=hadoop   IP=192.168.125.131      OPERATION=Submit Application Request    TARGET=ClientRMService  RESULT=SUCCESS  APPID=application_1563612848821_0006
2019-07-20 17:25:16,952 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: application_1563612848821_0006 State change from NEW to NEW_SAVING
2019-07-20 17:25:16,952 INFO org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore: Storing info for app: application_1563612848821_0006
2019-07-20 17:25:16,954 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: application_1563612848821_0006 State change from NEW_SAVING to SUBMITTED
2019-07-20 17:25:16,955 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: Application added - appId: application_1563612848821_0006 user: hadoop leaf-queue of parent: root #applications: 1
2019-07-20 17:25:16,955 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Accepted application application_1563612848821_0006 from user: hadoop, in queue: default
2019-07-20 17:25:16,955 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: application_1563612848821_0006 State change from SUBMITTED to ACCEPTED
2019-07-20 17:25:16,956 INFO org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService: Registering app attempt : appattempt_1563612848821_0006_000001
2019-07-20 17:25:16,956 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: appattempt_1563612848821_0006_000001 State change from NEW to SUBMITTED
2019-07-20 17:25:16,956 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: Application application_1563612848821_0006 from user: hadoop activated in queue: default
2019-07-20 17:25:16,956 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: Application added - appId: application_1563612848821_0006 user: org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue$User@7ccdead5, leaf-queue: default #user-pending-applications: 0 #user-active-applications: 1 #queue-pending-applications: 0 #queue-active-applications: 1
2019-07-20 17:25:16,956 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Added Application Attempt appattempt_1563612848821_0006_000001 to scheduler from user hadoop in queue default
2019-07-20 17:25:16,958 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: appattempt_1563612848821_0006_000001 State change from SUBMITTED to SCHEDULED
2019-07-20 17:25:17,020 INFO org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl: container_1563612848821_0006_01_000001 Container Transitioned from NEW to ALLOCATED
2019-07-20 17:25:17,020 INFO org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=hadoop   OPERATION=AM Allocated Container        TARGET=SchedulerApp     RESULT=SUCCESS  APPID=application_1563612848821_0006    CONTAINERID=container_1563612848821_0006_01_000001
2019-07-20 17:25:17,020 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerNode: Assigned container container_1563612848821_0006_01_000001 of capacity <memory:1024, vCores:1> on host mini3:40834, which has 1 containers, <memory:1024, vCores:1> used and <memory:7168, vCores:7> available after allocation
2019-07-20 17:25:17,020 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: assignedContainer application attempt=appattempt_1563612848821_0006_000001 container=Container: [ContainerId: container_1563612848821_0006_01_000001, NodeId: mini3:40834, NodeHttpAddress: mini3:8042, Resource: <memory:1024, vCores:1>, Priority: 0, Token: null, ] queue=default: capacity=1.0, absoluteCapacity=1.0, usedResources=<memory:0, vCores:0>, usedCapacity=0.0, absoluteUsedCapacity=0.0, numApps=1, numContainers=0 clusterResource=<memory:16384, vCores:16>
2019-07-20 17:25:17,020 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: Re-sorting assigned queue: root.default stats: default: capacity=1.0, absoluteCapacity=1.0, usedResources=<memory:1024, vCores:1>, usedCapacity=0.0625, absoluteUsedCapacity=0.0625, numApps=1, numContainers=1
2019-07-20 17:25:17,020 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: assignedContainer queue=root usedCapacity=0.0625 absoluteUsedCapacity=0.0625 used=<memory:1024, vCores:1> cluster=<memory:16384, vCores:16>
2019-07-20 17:25:17,021 INFO org.apache.hadoop.yarn.server.resourcemanager.security.NMTokenSecretManagerInRM: Sending NMToken for nodeId : mini3:40834 for container : container_1563612848821_0006_01_000001
2019-07-20 17:25:17,023 INFO org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl: container_1563612848821_0006_01_000001 Container Transitioned from ALLOCATED to ACQUIRED
2019-07-20 17:25:17,023 INFO org.apache.hadoop.yarn.server.resourcemanager.security.NMTokenSecretManagerInRM: Clear node set for appattempt_1563612848821_0006_000001
2019-07-20 17:25:17,023 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: Storing attempt: AppId: application_1563612848821_0006 AttemptId: appattempt_1563612848821_0006_000001 MasterContainer: Container: [ContainerId: container_1563612848821_0006_01_000001, NodeId: mini3:40834, NodeHttpAddress: mini3:8042, Resource: <memory:1024, vCores:1>, Priority: 0, Token: Token { kind: ContainerToken, service: 192.168.125.133:40834 }, ]
2019-07-20 17:25:17,023 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: appattempt_1563612848821_0006_000001 State change from SCHEDULED to ALLOCATED_SAVING
2019-07-20 17:25:17,024 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: appattempt_1563612848821_0006_000001 State change from ALLOCATED_SAVING to ALLOCATED
2019-07-20 17:25:17,024 INFO org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher: Launching masterappattempt_1563612848821_0006_000001
2019-07-20 17:25:17,026 INFO org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher: Setting up container Container: [ContainerId: container_1563612848821_0006_01_000001, NodeId: mini3:40834, NodeHttpAddress: mini3:8042, Resource: <memory:1024, vCores:1>, Priority: 0, Token: Token { kind: ContainerToken, service: 192.168.125.133:40834 }, ] for AM appattempt_1563612848821_0006_000001
2019-07-20 17:25:17,026 INFO org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher: Command to launch container container_1563612848821_0006_01_000001 : $JAVA_HOME/bin/java -Xmx424m  -Dlog.file="<LOG_DIR>/jobmanager.log" -Dlogback.configurationFile=file:logback.xml -Dlog4j.configuration=file:log4j.properties org.apache.flink.yarn.entrypoint.YarnSessionClusterEntrypoint  1> <LOG_DIR>/jobmanager.out 2> <LOG_DIR>/jobmanager.err
2019-07-20 17:25:17,026 INFO org.apache.hadoop.yarn.server.resourcemanager.security.AMRMTokenSecretManager: Create AMRMToken for ApplicationAttempt: appattempt_1563612848821_0006_000001
2019-07-20 17:25:17,026 INFO org.apache.hadoop.yarn.server.resourcemanager.security.AMRMTokenSecretManager: Creating password for appattempt_1563612848821_0006_000001
2019-07-20 17:25:17,066 INFO org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher: Done launching container Container: [ContainerId: container_1563612848821_0006_01_000001, NodeId: mini3:40834, NodeHttpAddress: mini3:8042, Resource: <memory:1024, vCores:1>, Priority: 0, Token: Token { kind: ContainerToken, service: 192.168.125.133:40834 }, ] for AM appattempt_1563612848821_0006_000001
2019-07-20 17:25:17,066 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: appattempt_1563612848821_0006_000001 State change from ALLOCATED to LAUNCHED
2019-07-20 17:25:18,132 INFO org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl: container_1563612848821_0006_01_000001 Container Transitioned from ACQUIRED to RUNNING
2019-07-20 17:25:23,485 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for appattempt_1563612848821_0006_000001 (auth:SIMPLE)
2019-07-20 17:25:23,502 INFO org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService: AM registration appattempt_1563612848821_0006_000001
2019-07-20 17:25:23,502 INFO org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=hadoop   IP=192.168.125.133      OPERATION=Register App Master   TARGET=ApplicationMasterService RESULT=SUCCESS  APPID=application_1563612848821_0006        APPATTEMPTID=appattempt_1563612848821_0006_000001
2019-07-20 17:25:23,502 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: appattempt_1563612848821_0006_000001 State change from LAUNCHED to RUNNING
2019-07-20 17:25:23,502 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: application_1563612848821_0006 State change from ACCEPTED to RUNNING
2019-07-20 17:25:24,388 INFO org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl: container_1563612848821_0006_01_000001 Container Transitioned from RUNNING to COMPLETED
2019-07-20 17:25:24,388 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.common.fica.FiCaSchedulerApp: Completed container: container_1563612848821_0006_01_000001 in state: COMPLETED event:FINISHED
2019-07-20 17:25:24,388 INFO org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=hadoop   OPERATION=AM Released Container TARGET=SchedulerApp     RESULT=SUCCESS  APPID=application_1563612848821_0006    CONTAINERID=container_1563612848821_0006_01_000001
2019-07-20 17:25:24,388 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerNode: Released container container_1563612848821_0006_01_000001 of capacity <memory:1024, vCores:1> on host mini3:40834, which currently has 0 containers, <memory:0, vCores:0> used and <memory:8192, vCores:8> available, release resources=true
2019-07-20 17:25:24,388 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: Updating application attempt appattempt_1563612848821_0006_000001 with final state: FAILED, and exit status: -103
2019-07-20 17:25:24,388 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: appattempt_1563612848821_0006_000001 State change from RUNNING to FINAL_SAVING
2019-07-20 17:25:24,388 INFO org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService: Unregistering app attempt : appattempt_1563612848821_0006_000001
2019-07-20 17:25:24,388 INFO org.apache.hadoop.yarn.server.resourcemanager.security.AMRMTokenSecretManager: Application finished, removing password for appattempt_1563612848821_0006_000001
2019-07-20 17:25:24,388 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: appattempt_1563612848821_0006_000001 State change from FINAL_SAVING to FAILED
2019-07-20 17:25:24,388 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: The number of failed attempts is 1. The max attempts is 1
2019-07-20 17:25:24,388 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: Updating application application_1563612848821_0006 with final state: FAILED
2019-07-20 17:25:24,388 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: application_1563612848821_0006 State change from RUNNING to FINAL_SAVING
2019-07-20 17:25:24,389 INFO org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher: Cleaning master appattempt_1563612848821_0006_000001
2019-07-20 17:25:24,388 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: default used=<memory:0, vCores:0> numContainers=0 user=hadoop user-resources=<memory:0, vCores:0>
2019-07-20 17:25:24,390 INFO org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore: Updating info for app: application_1563612848821_0006
2019-07-20 17:25:24,390 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: completedContainer container=Container: [ContainerId: container_1563612848821_0006_01_000001, NodeId: mini3:40834, NodeHttpAddress: mini3:8042, Resource: <memory:1024, vCores:1>, Priority: 0, Token: Token { kind: ContainerToken, service: 192.168.125.133:40834 }, ] queue=default: capacity=1.0, absoluteCapacity=1.0, usedResources=<memory:0, vCores:0>, usedCapacity=0.0, absoluteUsedCapacity=0.0, numApps=1, numContainers=0 cluster=<memory:16384, vCores:16>
2019-07-20 17:25:24,390 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: Application application_1563612848821_0006 failed 1 times due to AM Container for appattempt_1563612848821_0006_000001 exited with  exitCode: -103
For more detailed output, check application tracking page:http://mini1:8088/proxy/application_1563612848821_0006/Then, click on links to logs of each attempt.
Diagnostics: Container [pid=4258,containerID=container_1563612848821_0006_01_000001] is running beyond virtual memory limits. Current usage: 144.6 MB of 1 GB physical memory used; 2.2 GB of 2.1 GB virtual memory used. Killing container.
Dump of the process-tree for container_1563612848821_0006_01_000001 :
        |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
        |- 4272 4258 4258 4258 (java) 477 112 2228482048 36677 /app/jdk1.7/bin/java -Xmx424m -Dlog.file=/home/hadoop/apps/hadoop-2.6.4/logs/userlogs/application_1563612848821_0006/container_1563612848821_0006_01_000001/jobmanager.log -Dlogback.configurationFile=file:logback.xml -Dlog4j.configuration=file:log4j.properties org.apache.flink.yarn.entrypoint.YarnSessionClusterEntrypoint 
        |- 4258 4256 4258 4258 (bash) 3 2 108630016 330 /bin/bash -c /app/jdk1.7/bin/java -Xmx424m  -Dlog.file=/home/hadoop/apps/hadoop-2.6.4/logs/userlogs/application_1563612848821_0006/container_1563612848821_0006_01_000001/jobmanager.log -Dlogback.configurationFile=file:logback.xml -Dlog4j.configuration=file:log4j.properties org.apache.flink.yarn.entrypoint.YarnSessionClusterEntrypoint  1> /home/hadoop/apps/hadoop-2.6.4/logs/userlogs/application_1563612848821_0006/container_1563612848821_0006_01_000001/jobmanager.out 2> /home/hadoop/apps/hadoop-2.6.4/logs/userlogs/application_1563612848821_0006/container_1563612848821_0006_01_000001/jobmanager.err 

Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Failing this attempt. Failing the application.
2019-07-20 17:25:24,390 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: completedContainer queue=root usedCapacity=0.0 absoluteUsedCapacity=0.0 used=<memory:0, vCores:0> cluster=<memory:16384, vCores:16>
2019-07-20 17:25:24,390 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: application_1563612848821_0006 State change from FINAL_SAVING to FAILED
2019-07-20 17:25:24,391 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: Re-sorting completed queue: root.default stats: default: capacity=1.0, absoluteCapacity=1.0, usedResources=<memory:0, vCores:0>, usedCapacity=0.0, absoluteUsedCapacity=0.0, numApps=1, numContainers=0
2019-07-20 17:25:24,391 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Application attempt appattempt_1563612848821_0006_000001 released container container_1563612848821_0006_01_000001 on node: host: mini3:40834 #containers=0 available=8192 used=0 with event: FINISHED
2019-07-20 17:25:24,392 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Application Attempt appattempt_1563612848821_0006_000001 is done. finalState=FAILED
2019-07-20 17:25:24,392 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo: Application application_1563612848821_0006 requests cleared
2019-07-20 17:25:24,392 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: Application removed - appId: application_1563612848821_0006 user: hadoop queue: default #user-pending-applications: 0 #user-active-applications: 0 #queue-pending-applications: 0 #queue-active-applications: 0
2019-07-20 17:25:24,392 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: Application removed - appId: application_1563612848821_0006 user: hadoop leaf-queue of parent: root #applications: 0
2019-07-20 17:25:24,391 WARN org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=hadoop   OPERATION=Application Finished - Failed TARGET=RMAppManager     RESULT=FAILURE  DESCRIPTION=App failed with state: FAILED       PERMISSIONS=Application application_1563612848821_0006 failed 1 times due to AM Container for appattempt_1563612848821_0006_000001 exited with  exitCode: -103
For more detailed output, check application tracking page:http://mini1:8088/proxy/application_1563612848821_0006/Then, click on links to logs of each attempt.
Diagnostics: Container [pid=4258,containerID=container_1563612848821_0006_01_000001] is running beyond virtual memory limits. Current usage: 144.6 MB of 1 GB physical memory used; 2.2 GB of 2.1 GB virtual memory used. Killing container.
Dump of the process-tree for container_1563612848821_0006_01_000001 :
        |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
        |- 4272 4258 4258 4258 (java) 477 112 2228482048 36677 /app/jdk1.7/bin/java -Xmx424m -Dlog.file=/home/hadoop/apps/hadoop-2.6.4/logs/userlogs/application_1563612848821_0006/container_1563612848821_0006_01_000001/jobmanager.log -Dlogback.configurationFile=file:logback.xml -Dlog4j.configuration=file:log4j.properties org.apache.flink.yarn.entrypoint.YarnSessionClusterEntrypoint 
        |- 4258 4256 4258 4258 (bash) 3 2 108630016 330 /bin/bash -c /app/jdk1.7/bin/java -Xmx424m  -Dlog.file=/home/hadoop/apps/hadoop-2.6.4/logs/userlogs/application_1563612848821_0006/container_1563612848821_0006_01_000001/jobmanager.log -Dlogback.configurationFile=file:logback.xml -Dlog4j.configuration=file:log4j.properties org.apache.flink.yarn.entrypoint.YarnSessionClusterEntrypoint  1> /home/hadoop/apps/hadoop-2.6.4/logs/userlogs/application_1563612848821_0006/container_1563612848821_0006_01_000001/jobmanager.out 2> /home/hadoop/apps/hadoop-2.6.4/logs/userlogs/application_1563612848821_0006/container_1563612848821_0006_01_000001/jobmanager.err 

Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Failing this attempt. Failing the application.  APPID=application_1563612848821_0006
2019-07-20 17:25:24,393 INFO org.apache.hadoop.yarn.server.resourcemanager.RMAppManager$ApplicationSummary: appId=application_1563612848821_0006,name=test,user=hadoop,queue=default,state=FAILED,trackingUrl=http://mini1:8088/cluster/app/application_1563612848821_0006,appMasterHost=N/A,startTime=1563614716951,finishTime=1563614724388,finalStatus=FAILED
2019-07-20 17:29:14,708 INFO org.apache.hadoop.yarn.server.resourcemanager.ClientRMService: Allocated new applicationId: 7
2019-07-20 17:29:17,436 INFO org.apache.hadoop.yarn.server.resourcemanager.ClientRMService: Application with id 7 submitted by user hadoop
2019-07-20 17:29:17,436 INFO org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=hadoop   IP=192.168.125.131      OPERATION=Submit Application Request    TARGET=ClientRMService  RESULT=SUCCESS  APPID=application_1563612848821_0007
2019-07-20 17:29:17,437 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: Storing application with id application_1563612848821_0007
2019-07-20 17:29:17,437 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: application_1563612848821_0007 State change from NEW to NEW_SAVING
2019-07-20 17:29:17,437 INFO org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore: Storing info for app: application_1563612848821_0007
2019-07-20 17:29:17,437 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: application_1563612848821_0007 State change from NEW_SAVING to SUBMITTED
2019-07-20 17:29:17,439 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: Application added - appId: application_1563612848821_0007 user: hadoop leaf-queue of parent: root #applications: 1
2019-07-20 17:29:17,439 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Accepted application application_1563612848821_0007 from user: hadoop, in queue: default
2019-07-20 17:29:17,440 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: application_1563612848821_0007 State change from SUBMITTED to ACCEPTED
2019-07-20 17:29:17,441 INFO org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService: Registering app attempt : appattempt_1563612848821_0007_000001
2019-07-20 17:29:17,441 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: appattempt_1563612848821_0007_000001 State change from NEW to SUBMITTED
2019-07-20 17:29:17,452 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: Application application_1563612848821_0007 from user: hadoop activated in queue: default
2019-07-20 17:29:17,452 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: Application added - appId: application_1563612848821_0007 user: org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue$User@6828a6e2, leaf-queue: default #user-pending-applications: 0 #user-active-applications: 1 #queue-pending-applications: 0 #queue-active-applications: 1
2019-07-20 17:29:17,452 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Added Application Attempt appattempt_1563612848821_0007_000001 to scheduler from user hadoop in queue default
2019-07-20 17:29:17,453 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: appattempt_1563612848821_0007_000001 State change from SUBMITTED to SCHEDULED
2019-07-20 17:29:17,992 INFO org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl: container_1563612848821_0007_01_000001 Container Transitioned from NEW to ALLOCATED
2019-07-20 17:29:17,992 INFO org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=hadoop   OPERATION=AM Allocated Container        TARGET=SchedulerApp     RESULT=SUCCESS  APPID=application_1563612848821_0007    CONTAINERID=container_1563612848821_0007_01_000001
2019-07-20 17:29:17,992 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerNode: Assigned container container_1563612848821_0007_01_000001 of capacity <memory:1024, vCores:1> on host mini2:57152, which has 1 containers, <memory:1024, vCores:1> used and <memory:7168, vCores:7> available after allocation
2019-07-20 17:29:17,992 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: assignedContainer application attempt=appattempt_1563612848821_0007_000001 container=Container: [ContainerId: container_1563612848821_0007_01_000001, NodeId: mini2:57152, NodeHttpAddress: mini2:8042, Resource: <memory:1024, vCores:1>, Priority: 0, Token: null, ] queue=default: capacity=1.0, absoluteCapacity=1.0, usedResources=<memory:0, vCores:0>, usedCapacity=0.0, absoluteUsedCapacity=0.0, numApps=1, numContainers=0 clusterResource=<memory:16384, vCores:16>
2019-07-20 17:29:17,992 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: Re-sorting assigned queue: root.default stats: default: capacity=1.0, absoluteCapacity=1.0, usedResources=<memory:1024, vCores:1>, usedCapacity=0.0625, absoluteUsedCapacity=0.0625, numApps=1, numContainers=1
2019-07-20 17:29:17,992 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: assignedContainer queue=root usedCapacity=0.0625 absoluteUsedCapacity=0.0625 used=<memory:1024, vCores:1> cluster=<memory:16384, vCores:16>
2019-07-20 17:29:17,994 INFO org.apache.hadoop.yarn.server.resourcemanager.security.NMTokenSecretManagerInRM: Sending NMToken for nodeId : mini2:57152 for container : container_1563612848821_0007_01_000001
2019-07-20 17:29:17,996 INFO org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl: container_1563612848821_0007_01_000001 Container Transitioned from ALLOCATED to ACQUIRED
2019-07-20 17:29:17,996 INFO org.apache.hadoop.yarn.server.resourcemanager.security.NMTokenSecretManagerInRM: Clear node set for appattempt_1563612848821_0007_000001
2019-07-20 17:29:17,996 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: Storing attempt: AppId: application_1563612848821_0007 AttemptId: appattempt_1563612848821_0007_000001 MasterContainer: Container: [ContainerId: container_1563612848821_0007_01_000001, NodeId: mini2:57152, NodeHttpAddress: mini2:8042, Resource: <memory:1024, vCores:1>, Priority: 0, Token: Token { kind: ContainerToken, service: 192.168.125.132:57152 }, ]
2019-07-20 17:29:17,996 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: appattempt_1563612848821_0007_000001 State change from SCHEDULED to ALLOCATED_SAVING
2019-07-20 17:29:17,997 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: appattempt_1563612848821_0007_000001 State change from ALLOCATED_SAVING to ALLOCATED
2019-07-20 17:29:17,997 INFO org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher: Launching masterappattempt_1563612848821_0007_000001
2019-07-20 17:29:18,002 INFO org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher: Setting up container Container: [ContainerId: container_1563612848821_0007_01_000001, NodeId: mini2:57152, NodeHttpAddress: mini2:8042, Resource: <memory:1024, vCores:1>, Priority: 0, Token: Token { kind: ContainerToken, service: 192.168.125.132:57152 }, ] for AM appattempt_1563612848821_0007_000001
2019-07-20 17:29:18,002 INFO org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher: Command to launch container container_1563612848821_0007_01_000001 : $JAVA_HOME/bin/java -Xmx424m  -Dlog.file="<LOG_DIR>/jobmanager.log" -Dlogback.configurationFile=file:logback.xml -Dlog4j.configuration=file:log4j.properties org.apache.flink.yarn.entrypoint.YarnSessionClusterEntrypoint  1> <LOG_DIR>/jobmanager.out 2> <LOG_DIR>/jobmanager.err
2019-07-20 17:29:18,002 INFO org.apache.hadoop.yarn.server.resourcemanager.security.AMRMTokenSecretManager: Create AMRMToken for ApplicationAttempt: appattempt_1563612848821_0007_000001
2019-07-20 17:29:18,002 INFO org.apache.hadoop.yarn.server.resourcemanager.security.AMRMTokenSecretManager: Creating password for appattempt_1563612848821_0007_000001
2019-07-20 17:29:18,031 INFO org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher: Done launching container Container: [ContainerId: container_1563612848821_0007_01_000001, NodeId: mini2:57152, NodeHttpAddress: mini2:8042, Resource: <memory:1024, vCores:1>, Priority: 0, Token: Token { kind: ContainerToken, service: 192.168.125.132:57152 }, ] for AM appattempt_1563612848821_0007_000001
2019-07-20 17:29:18,031 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: appattempt_1563612848821_0007_000001 State change from ALLOCATED to LAUNCHED
2019-07-20 17:29:19,374 INFO org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl: container_1563612848821_0007_01_000001 Container Transitioned from ACQUIRED to RUNNING
2019-07-20 17:29:26,762 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for appattempt_1563612848821_0007_000001 (auth:SIMPLE)
2019-07-20 17:29:26,785 INFO org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService: AM registration appattempt_1563612848821_0007_000001
2019-07-20 17:29:26,785 INFO org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=hadoop   IP=192.168.125.132      OPERATION=Register App Master   TARGET=ApplicationMasterService RESULT=SUCCESS  APPID=application_1563612848821_0007        APPATTEMPTID=appattempt_1563612848821_0007_000001
2019-07-20 17:29:26,785 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: appattempt_1563612848821_0007_000001 State change from LAUNCHED to RUNNING
2019-07-20 17:29:26,785 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: application_1563612848821_0007 State change from ACCEPTED to RUNNING
2019-07-20 17:29:29,334 INFO org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl: container_1563612848821_0007_01_000001 Container Transitioned from RUNNING to COMPLETED
2019-07-20 17:29:29,334 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.common.fica.FiCaSchedulerApp: Completed container: container_1563612848821_0007_01_000001 in state: COMPLETED event:FINISHED
2019-07-20 17:29:29,334 INFO org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=hadoop   OPERATION=AM Released Container TARGET=SchedulerApp     RESULT=SUCCESS  APPID=application_1563612848821_0007    CONTAINERID=container_1563612848821_0007_01_000001
2019-07-20 17:29:29,334 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerNode: Released container container_1563612848821_0007_01_000001 of capacity <memory:1024, vCores:1> on host mini2:57152, which currently has 0 containers, <memory:0, vCores:0> used and <memory:8192, vCores:8> available, release resources=true
2019-07-20 17:29:29,334 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: Updating application attempt appattempt_1563612848821_0007_000001 with final state: FAILED, and exit status: -103
2019-07-20 17:29:29,334 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: appattempt_1563612848821_0007_000001 State change from RUNNING to FINAL_SAVING
2019-07-20 17:29:29,334 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: default used=<memory:0, vCores:0> numContainers=0 user=hadoop user-resources=<memory:0, vCores:0>
2019-07-20 17:29:29,334 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: completedContainer container=Container: [ContainerId: container_1563612848821_0007_01_000001, NodeId: mini2:57152, NodeHttpAddress: mini2:8042, Resource: <memory:1024, vCores:1>, Priority: 0, Token: Token { kind: ContainerToken, service: 192.168.125.132:57152 }, ] queue=default: capacity=1.0, absoluteCapacity=1.0, usedResources=<memory:0, vCores:0>, usedCapacity=0.0, absoluteUsedCapacity=0.0, numApps=1, numContainers=0 cluster=<memory:16384, vCores:16>
2019-07-20 17:29:29,334 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: completedContainer queue=root usedCapacity=0.0 absoluteUsedCapacity=0.0 used=<memory:0, vCores:0> cluster=<memory:16384, vCores:16>
2019-07-20 17:29:29,334 INFO org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService: Unregistering app attempt : appattempt_1563612848821_0007_000001
2019-07-20 17:29:29,334 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: Re-sorting completed queue: root.default stats: default: capacity=1.0, absoluteCapacity=1.0, usedResources=<memory:0, vCores:0>, usedCapacity=0.0, absoluteUsedCapacity=0.0, numApps=1, numContainers=0
2019-07-20 17:29:29,335 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Application attempt appattempt_1563612848821_0007_000001 released container container_1563612848821_0007_01_000001 on node: host: mini2:57152 #containers=0 available=8192 used=0 with event: FINISHED
2019-07-20 17:29:29,335 INFO org.apache.hadoop.yarn.server.resourcemanager.security.AMRMTokenSecretManager: Application finished, removing password for appattempt_1563612848821_0007_000001
2019-07-20 17:29:29,335 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: appattempt_1563612848821_0007_000001 State change from FINAL_SAVING to FAILED
2019-07-20 17:29:29,335 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: The number of failed attempts is 1. The max attempts is 1
2019-07-20 17:29:29,335 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: Updating application application_1563612848821_0007 with final state: FAILED
2019-07-20 17:29:29,335 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: application_1563612848821_0007 State change from RUNNING to FINAL_SAVING
2019-07-20 17:29:29,335 INFO org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore: Updating info for app: application_1563612848821_0007
2019-07-20 17:29:29,336 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Application Attempt appattempt_1563612848821_0007_000001 is done. finalState=FAILED
2019-07-20 17:29:29,336 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo: Application application_1563612848821_0007 requests cleared
2019-07-20 17:29:29,336 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: Application removed - appId: application_1563612848821_0007 user: hadoop queue: default #user-pending-applications: 0 #user-active-applications: 0 #queue-pending-applications: 0 #queue-active-applications: 0
2019-07-20 17:29:29,336 INFO org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher: Cleaning master appattempt_1563612848821_0007_000001
2019-07-20 17:29:29,337 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: Application application_1563612848821_0007 failed 1 times due to AM Container for appattempt_1563612848821_0007_000001 exited with  exitCode: -103
For more detailed output, check application tracking page:http://mini1:8088/proxy/application_1563612848821_0007/Then, click on links to logs of each attempt.
Diagnostics: Container [pid=5351,containerID=container_1563612848821_0007_01_000001] is running beyond virtual memory limits. Current usage: 152.0 MB of 1 GB physical memory used; 2.2 GB of 2.1 GB virtual memory used. Killing container.
Dump of the process-tree for container_1563612848821_0007_01_000001 :
        |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
        |- 5351 5349 5351 5351 (bash) 0 0 108646400 295 /bin/bash -c /app/jdk1.7/bin/java -Xmx424m  -Dlog.file=/home/hadoop/apps/hadoop-2.6.4/logs/userlogs/application_1563612848821_0007/container_1563612848821_0007_01_000001/jobmanager.log -Dlogback.configurationFile=file:logback.xml -Dlog4j.configuration=file:log4j.properties org.apache.flink.yarn.entrypoint.YarnSessionClusterEntrypoint  1> /home/hadoop/apps/hadoop-2.6.4/logs/userlogs/application_1563612848821_0007/container_1563612848821_0007_01_000001/jobmanager.out 2> /home/hadoop/apps/hadoop-2.6.4/logs/userlogs/application_1563612848821_0007/container_1563612848821_0007_01_000001/jobmanager.err 
        |- 5365 5351 5351 5351 (java) 550 121 2236710912 38608 /app/jdk1.7/bin/java -Xmx424m -Dlog.file=/home/hadoop/apps/hadoop-2.6.4/logs/userlogs/application_1563612848821_0007/container_1563612848821_0007_01_000001/jobmanager.log -Dlogback.configurationFile=file:logback.xml -Dlog4j.configuration=file:log4j.properties org.apache.flink.yarn.entrypoint.YarnSessionClusterEntrypoint 

Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Failing this attempt. Failing the application.
2019-07-20 17:29:29,338 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: application_1563612848821_0007 State change from FINAL_SAVING to FAILED
2019-07-20 17:29:29,338 WARN org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=hadoop   OPERATION=Application Finished - Failed TARGET=RMAppManager     RESULT=FAILURE  DESCRIPTION=App failed with state: FAILED       PERMISSIONS=Application application_1563612848821_0007 failed 1 times due to AM Container for appattempt_1563612848821_0007_000001 exited with  exitCode: -103
For more detailed output, check application tracking page:http://mini1:8088/proxy/application_1563612848821_0007/Then, click on links to logs of each attempt.
Diagnostics: Container [pid=5351,containerID=container_1563612848821_0007_01_000001] is running beyond virtual memory limits. Current usage: 152.0 MB of 1 GB physical memory used; 2.2 GB of 2.1 GB virtual memory used. Killing container.
Dump of the process-tree for container_1563612848821_0007_01_000001 :
        |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
        |- 5351 5349 5351 5351 (bash) 0 0 108646400 295 /bin/bash -c /app/jdk1.7/bin/java -Xmx424m  -Dlog.file=/home/hadoop/apps/hadoop-2.6.4/logs/userlogs/application_1563612848821_0007/container_1563612848821_0007_01_000001/jobmanager.log -Dlogback.configurationFile=file:logback.xml -Dlog4j.configuration=file:log4j.properties org.apache.flink.yarn.entrypoint.YarnSessionClusterEntrypoint  1> /home/hadoop/apps/hadoop-2.6.4/logs/userlogs/application_1563612848821_0007/container_1563612848821_0007_01_000001/jobmanager.out 2> /home/hadoop/apps/hadoop-2.6.4/logs/userlogs/application_1563612848821_0007/container_1563612848821_0007_01_000001/jobmanager.err 
        |- 5365 5351 5351 5351 (java) 550 121 2236710912 38608 /app/jdk1.7/bin/java -Xmx424m -Dlog.file=/home/hadoop/apps/hadoop-2.6.4/logs/userlogs/application_1563612848821_0007/container_1563612848821_0007_01_000001/jobmanager.log -Dlogback.configurationFile=file:logback.xml -Dlog4j.configuration=file:log4j.properties org.apache.flink.yarn.entrypoint.YarnSessionClusterEntrypoint

2.没有启动Hadoop

解决方案，先启动hadoop，运行yarn-session.sh

2019-07-20 17:56:41,823 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: jobmanager.rpc.address, mini1
2019-07-20 17:56:41,824 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: jobmanager.rpc.port, 6123
2019-07-20 17:56:41,824 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: jobmanager.heap.size, 1024m
2019-07-20 17:56:41,825 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: taskmanager.heap.size, 1024m
2019-07-20 17:56:41,825 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: taskmanager.numberOfTaskSlots, 1
2019-07-20 17:56:41,825 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: parallelism.default, 1
2019-07-20 17:56:41,825 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: rest.port, 8081
2019-07-20 17:56:41,840 INFO  org.apache.flink.yarn.cli.FlinkYarnSessionCli                 - Found Yarn properties file under /tmp/.yarn-properties-hadoop.
2019-07-20 17:56:42,399 WARN  org.apache.hadoop.util.NativeCodeLoader                       - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2019-07-20 17:56:42,507 INFO  org.apache.flink.runtime.security.modules.HadoopModule        - Hadoop user set to hadoop (auth:SIMPLE)
2019-07-20 17:56:42,607 INFO  org.apache.hadoop.yarn.client.RMProxy                         - Connecting to ResourceManager at mini1/192.168.125.131:8032
2019-07-20 17:56:42,792 INFO  org.apache.flink.yarn.cli.FlinkYarnSessionCli                 - The argument n is deprecated in will be ignored.
2019-07-20 17:56:43,853 INFO  org.apache.hadoop.ipc.Client                                  - Retrying connect to server: mini1/192.168.125.131:8032. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2019-07-20 17:56:44,854 INFO  org.apache.hadoop.ipc.Client                                  - Retrying connect to server: mini1/192.168.125.131:8032. Already tried 1 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2019-07-20 17:56:45,855 INFO  org.apache.hadoop.ipc.Client                                  - Retrying connect to server: mini1/192.168.125.131:8032. Already tried 2 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2019-07-20 17:56:46,856 INFO  org.apache.hadoop.ipc.Client                                  - Retrying connect to server: mini1/192.168.125.131:8032. Already tried 3 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2019-07-20 17:56:47,857 INFO  org.apache.hadoop.ipc.Client                                  - Retrying connect to server: mini1/192.168.125.131:8032. Already tried 4 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2019-07-20 17:56:48,858 INFO  org.apache.hadoop.ipc.Client                                  - Retrying connect to server: mini1/192.168.125.131:8032. Already tried 5 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2019-07-20 17:56:49,859 INFO  org.apache.hadoop.ipc.Client                                  - Retrying connect to server: mini1/192.168.125.131:8032. Already tried 6 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2019-07-20 17:56:50,860 INFO  org.apache.hadoop.ipc.Client                                  - Retrying connect to server: mini1/192.168.125.131:8032. Already tried 7 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)

 

小柠猿
关注
专栏目录
flink on yarn的会话模式中yarn-session.sh 参数
JustinMars的博客
07-19 1245
是 Apache Flink 在 YARN（Hadoop 的资源管理系统）上启动 Flink 会话（Session）的脚本。通过这个脚本，你可以在 YARN 上提交 Flink 任务，并在一个长时间运行的会话中执行多个作业。: 指定要在 YARN 上启动的 TaskManager 容器数量，也就是执行 Flink 任务的资源数量。: 指定 JobManager 容器的内存大小，以MB为单位。: 指定每个 TaskManager 容器的内存大小，以MB为单位。: 启动作业时可以指定特定的应用程序选项。
Flink启动yarn-session报错：Error while running the Flink session
AlianBlank的博客
09-04 265
总结一下，当在使用Flink启动yarn-session时遇到"Error while running the Flink session. java.lang.NoSuchMethodError"的报错时，首先需要确保Flink版本与依赖项版本兼容。在使用Flink启动yarn-session时，有时会遇到"Error while running the Flink session. java.lang.NoSuchMethodError"的报错。在实际使用时，你需要根据自己的需求进行相应的配置。
[YARN] 一个JDK的bug导致RM无法分配Container
张伟的专栏
09-19 4323
[YARN] 一个JDK的bug导致RM无法分配Container 一月 20, 2017 0 条评论 最近集群因为NM的OOM，然后决定把ContainerMetric给关闭了，然后采取了批量重启NM的方式，采取的步骤是先批量下线，然后在上线，后来发现集群任务越来越慢，集群的利用率越来越低 以下是集群可用内存，一直在增加 其中出现的现象是： 1.事件队列队列堆积非常严重，最高500W+...
Flink:Yarn-Session 启动
u012566690的专栏
08-29 2937
背景： 自己搭建了flink 单机版本时，到 flink on yarn 这一步卡住了，不能正常运行执行./bin/yarn-session.sh -n 4 -jm 1024m -tm 4096m 包括 GetClusterMetricsRequestProto cannot be cast to com.google.protobuf.Message 主要原因是未指定HADOOP_CONF_DIR环境变量，下面给出整个过程 ##1. 参照文档，下载&&编译flink 源码,这个时候，s
flink on yarn启动session 指定队列
weixin_31437187的博客
06-28 152
配置相关视频讲解：C语言程序设计入门之环境安装Flink on Yarn启动Session指定队列教程 一、整体流程 下面是启动Flink on Yarn Session并指定队列的步骤： 步骤 描述 1 准备Flink程序jar包和配置文件 2 启动Yarn集群 3 提交Flink on Yarn...
通过Flink YARN Session来运行任务
zg_hover的专栏
10-11 4817
通过Flink YARN Session来运行任务 本文先介绍Flink YARN Session的基本概念，并介绍如何启动Flink Session，并通过它来运行任务。 简介 Apache Hadoop YARN是一个集群资源管理框架。可以在它之上运行各种分布式应用。在其他应用运行时，Flink也可以运行在其上。 在Flink YARN Session启动时会按配置或参数来申请资源，并会启动Flink的基础服务：JobManager和TaskManagers。这样，就可以向它提交Flink应用了；每个s
1.flink的yarn-session环境
流月up的博客
10-18 459
在hadoop环境安装好之后,需要一个yarn-session环境来跑flink任务,此篇任务是在hadoop yarn环境上部署一个flink yarn-session环境,flink 1.17.1版本相关文章聚合链接。
flink yarn-session.sh启动任务指定的应用名和队列无效问题
云想慕尘的专栏
07-01 4914
问题背景 我有一个hadoop集群和一台机器A. 我在机器A上，使用命令: bin/yarn-session.sh -n 5 -jm 1g -tm 4g -s 4 -qu root.flink -nm fsql-cli > /dev/null 2>&1 & 启动了一个yarn-session，其中，指定了队列root.flink和应用名fsql-cli。 但是，在yarn控制台中看到应用的名称和队列跟指定的名称大多数情况下不一致，如图： 启动后，应用名为：Flink sess
yarn-session下的flink应用的提交与关闭
JustinMars的博客
07-26 965
flink应用的提交与关闭
Flink on yarn运行的Session-Cluster和Per-Job-Cluster模式详解
最新发布
sunxunyong的博客
07-08 399
Attached模式下，Flink Client创建完集群后，不能关闭命令行窗口，需要与集群之间维持连接，好处是能够感知集群的退出，集群退出之后有机会做一些资源清理等动作，此处的清理是Flink作业可能占用外部的资源。bin/flink run -d -m yarn-cluster #-d（表示 detached 模式） -m yarn-cluster（表示指定 YARN 集群模式） 启动per-job运行模式。在向Flink集群提交Job的时候, 如果资源被用完了,则新的Job不能正常提交。
yarn-session.sh 错误解决
11-19 1865
$FLINK_HOME/bin/yarn-session.sh -n 3 -s 2 -jm 2048 -tm 1024 执行这个命令 报错缺少jar报 flink1.2.1缺一个，flinK.1.2.5缺两个 flink-shaded-hadoop-2-uber-2.7.2-10.0.jar下载 如果不能用 自己编译代码。确实有点麻烦 第一个错误 缺少命令 【问题与解决】Flink启动时报错：NoSuchMethodError: org.apache.commons.cli.Op..
3.Flink-On-Yarn开发使用\原理\Session会话模式\Per-Job模式
涂作权的博客
07-06 1926
本文来自：Flink1.12-2021黑马程序员贺岁视频3.Flink-On-Yarn开发使用 3.1.原理 3.2.两种模式 3.2.1.Session会话模式 3.2.2.Per-Job模式 3.3.操作 3.4.测试 3.4.1.Session会话模式 3.4.2.PerJob模式为什么使用flink on yarn ? 在实际开发中，使用Flink时，更多的使用方式是Flink On Yarn模式，原因如下：具体步骤： 准备Flink程序jar和配置 0：提交 1：上传任务/依赖jar,配置到hdf
Flink（五十一）—— 一张图轻松掌握 Flink on YARN 基础架构与启动流程
weixin_42073629的博客
10-22 441
Flink 支持 Standalone 独立部署和 YARN、Kubernetes、Mesos 等集群部署模式，其中 YARN 集群部署模式在国内的应用越来越广泛。Flink 社区将推出 Flink on YARN 应用解读系列文章，分为上、下两篇。本文基于 FLIP-6 重构后的资源调度模型将介绍 Flink on YARN 应用启动全流程，并进行详细步骤解析。 Flink on YARN 应用启动流程图 Flink on YARN 集群部署模式涉及 YARN 和 Flink 两大开源框架，应用启动流
flink提交作业到yarn
chanyue123的博客
12-01 8578
讲解flink如何提交到yarn
大数据flink篇之三-flink运行环境安装后续一yarn-session安装
a18852867035的博客
10-14 1126
具体的，在生产环境中，flink一般会交由yarn、k8s等资源管理平台来处理。本章主要讲解yarn模式下的session cluster模式。Hadoop 必須保证在 2.2 以上，且必須裝有 hdfs 服务。Hadoop安装后续会有相关说明。这种模式所有的job都共享一个jobmanager，启动的集群只有一个jobmanager。首先确保在/etc/profile中添加了配置项。
Flink提交作业到yarn session集群main函数命令行参数传递
weixin_43525585的博客
12-18 3142
创建flink的yarn session集群 yarn-session.sh -n 3 -s 3 -nm flink-sessiontest -d -q yarn-session选项： -n,–container ：在yarn中启动container的个数，实质就是TaskManager的个数 -s,–slots ：每个TaskManager管理的Slot个数 -nm,–name :给当前的yarn-session(Flink集群)起一个名字 -d,–detached:后台独立模式启动，守护进程 -tm,
[Flink] Flink On Yarn（yarn-session.sh）启动错误
雨潇的专栏
06-05 2967
在Flink上启动 yarn-session.sh时出现 The number of requested virtual cores for application master 1 exceeds the maximum number of virtual cores 0 available in the Yarn Cluster.错误
yarn-session.sh启动报错的六种解决问题集合包括完全分布式和伪分布式
weixin_52810349的博客
06-08 860
VERSION里的datanodeUuid一致导致的冲突，这里是因为从节点和主节点用一样的datanodeUuid导致node块在yarn集群里冲突了。第二种hdfs的块坏了（yarn-session.sh其实是运行多次没有运行成功，发送过去坏了没有删除）第三种内存爆了，电脑满90%以上内存了，可以调调虚拟机内存配比。在 yarn-site.xml。在yarn-site.xml。在hdfs-site.xml。调成5个，设置为5就可以了。从节点的和主节点一致才能运行。第四种备份副本块的问题。
Flink1.10从0到1：（四）Flink On Yarn 和命令详解
joshua
07-07 752
一、简介 Flink on Yarn 模式的原理是依靠 YARN 来调度 Flink 任务，目前在企业中使用较多。这种模式的好处是可以充分利用集群资源，提高集群机器的利用率，并且只需要 1 套 Hadoop集群，就可以执行 MapReduce 和 Spark 任务，还可以执行 Flink 任务等，操作非常方便，不需要维护多套集群，运维方面也很轻松。Flink on Yarn 模式需要依赖 Hadoop 集群，并且Hadoop 的版本需要是 2.2 及以上。 PS：本人运行环境是基于clo...
FLink yarn模式
06-03
Flink YARN模式是在YARN集群上运行Flink任务的一种模式。使用YARN模式可以更好地利用集群资源，提高任务的并发度和可扩展性。Flink YARN模式的部署可以分为以下几个步骤： 1. 准备YARN环境：首先需要在YARN集群中准备好Java环境和Flink的安装包。可以使用Hadoop的命令行工具hdfs dfs来上传Flink安装包到HDFS中。 2. 配置Flink：编辑Flink的配置文件，设置Flink集群的参数，如TaskManager数量、内存等。同时需要配置Flink的YARN相关参数，如yarn.application.name、yarn.queue、yarn.container.memory等。 3. 提交Flink任务：使用Flink提供的yarn-session.sh脚本来启动Flink YARN Session，将Flink任务提交到YARN集群中运行。启动命令如下： ``` ./bin/yarn-session.sh -n 3 -jm 1024m -tm 1024m ``` 其中-n参数指定TaskManager的数量，-jm和-tm参数分别指定JobManager和TaskManager的内存大小。 4. 监控Flink任务：使用Flink Web Dashboard或YARN的ResourceManager Web UI来监控Flink任务的执行情况。可以查看任务的运行状态、日志信息、资源使用情况等。 需要注意的是，在Flink YARN模式中，Flink任务的运行依赖于YARN集群的资源管理，因此需要根据集群的实际情况来配置Flink的YARN参数。同时，由于YARN集群的资源是动态变化的，因此在任务运行过程中可能需要调整Flink集群的配置和参数，以更好地适应集群资源的变化。
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值