【奇奇怪怪bug】Flink 1.10 on yarn ,application 在yarn 显示Running ，webui 显示Failed

最新推荐文章于 2023-03-29 15:31:53 发布
置顶 最新推荐文章于 2023-03-29 15:31:53 发布
阅读量2.8k 收藏 8
点赞数 3
分类专栏： Flink 文章标签： flink 1.10 yarn
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/Zsigner/article/details/108128023
版权

 今天产品气势汹汹的跑过来跟我说，生产环境的实时程序不对啊！！！！添加数据一直看不到展示

开始不可能三连： 1.不可能吧，早上还好端端的

                              2.不可能会出问题的，可能是网络延迟

                               3.不可能出不来啊，是不是你没清缓存，你清缓存试试看

打发了产品回去之后，立马登上去看要不然要祭旗了。。。

1.排查


 1.1 yarn 运行的好端端的,没收到电话跟短信是正常的（ps：这里加了监控实时任务挂掉的话是会电话短信通知，呼爆你的。。。。）

 1.2 看看JobManager 有没有挂，webui正常打开，我的天居然在Running job上没有看到任务，看来这锅是要背起来了，左右看看能不能把这锅甩出去.....

 1.3 因为已经挂了在webui点不开log【error:Internal server error.】

 1.4 关键时刻还是得靠yarn老大哥的log

2020-08-20 09:22:52,687 INFO  org.apache.flink.yarn.YarnResourceManager                     - The heartbeat of TaskManager with id container_e14_1594608422123_4297_01_000002 timed out.
2020-08-20 09:22:52,688 INFO  org.apache.flink.yarn.YarnResourceManager                     - Closing TaskExecutor connection container_e14_1594608422123_4297_01_000002 because: The heartbeat of TaskManager with id container_e14_1594608422123_4297_01_000002  timed out.
2020-08-20 09:22:52,692 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source: Custom Source -> (Sink: Print to Std. Out, Map -> Filter) (1/3) (cbe68593452a4ede5106d642b57c5b4d) switched from RUNNING to FAILED.
java.util.concurrent.TimeoutException: The heartbeat of TaskManager with id container_e14_1594608422123_4297_01_000002  timed out.
	at org.apache.flink.runtime.resourcemanager.ResourceManager$TaskManagerHeartbeatListener.notifyHeartbeatTimeout(ResourceManager.java:1125)
	at org.apache.flink.runtime.heartbeat.HeartbeatMonitorImpl.run(HeartbeatMonitorImpl.java:109)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRunAsync(AkkaRpcActor.java:397)
	at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:190)
	at org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:74)
	at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:152)
	at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26)
	at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21)
	at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123)
	at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21)
	at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:170)
	at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
	at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
	at akka.actor.Actor$class.aroundReceive(Actor.scala:517)
	at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225)
	at akka.actor.ActorCell.receiveMessage(ActorCell.scala:592)
	at akka.actor.ActorCell.invoke(ActorCell.scala:561)
	at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258)
	at akka.dispatch.Mailbox.run(Mailbox.scala:225)
	at akka.dispatch.Mailbox.exec(Mailbox.scala:235)
	at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
	at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
	at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
	at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
2020-08-20 09:22:52,695 INFO  org.apache.flink.runtime.executiongraph.failover.flip1.RestartPipelinedRegionStrategy  - Calculating tasks to restart to recover the failed task e3dfc0d7e9ecd8a43f85f0b68ebf3b80_0.
2020-08-20 09:22:52,695 INFO  org.apache.flink.runtime.executiongraph.failover.flip1.RestartPipelinedRegionStrategy  - 1 tasks should be restarted to recover the failed task e3dfc0d7e9ecd8a43f85f0b68ebf3b80_0. 
2020-08-20 09:22:52,696 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Job Label2ES Streaming  (3dc5e025bf4569cf32f17438317a13d1) switched from state RUNNING to FAILING.
org.apache.flink.runtime.JobException: Recovery is suppressed by NoRestartBackoffTimeStrategy
	at org.apache.flink.runtime.executiongraph.failover.flip1.ExecutionFailureHandler.handleFailure(ExecutionFailureHandler.java:110)
	at org.apache.flink.runtime.executiongraph.failover.flip1.ExecutionFailureHandler.getFailureHandlingResult(ExecutionFailureHandler.java:76)
	at org.apache.flink.runtime.scheduler.DefaultScheduler.handleTaskFailure(DefaultScheduler.java:192)
	at org.apache.flink.runtime.scheduler.DefaultScheduler.maybeHandleTaskFailure(DefaultScheduler.java:186)
	at org.apache.flink.runtime.scheduler.DefaultScheduler.updateTaskExecutionStateInternal(DefaultScheduler.java:180)
	at org.apache.flink.runtime.scheduler.SchedulerBase.updateTaskExecutionState(SchedulerBase.java:484)
	at org.apache.flink.runtime.scheduler.UpdateSchedulerNgOnInternalFailuresListener.notifyTaskFailure(UpdateSchedulerNgOnInternalFailuresListener.java:49)
	at org.apache.flink.runtime.executiongraph.ExecutionGraph.notifySchedulerNgAboutInternalTaskFailure(ExecutionGraph.java:1703)
	at org.apache.flink.runtime.executiongraph.Execution.processFail(Execution.java:1252)
	at org.apache.flink.runtime.executiongraph.Execution.processFail(Execution.java:1220)
	at org.apache.flink.runtime.executiongraph.Execution.fail(Execution.java:955)
	at org.apache.flink.runtime.jobmaster.slotpool.SingleLogicalSlot.signalPayloadRelease(SingleLogicalSlot.java:173)
	at org.apache.flink.runtime.jobmaster.slotpool.SingleLogicalSlot.release(SingleLogicalSlot.java:165)
	at org.apache.flink.runtime.jobmaster.slotpool.SlotSharingManager$SingleTaskSlot.release(SlotSharingManager.java:732)
	at org.apache.flink.runtime.jobmaster.slotpool.SlotSharingManager$MultiTaskSlot.release(SlotSharingManager.java:537)
	at org.apache.flink.runtime.jobmaster.slotpool.AllocatedSlot.releasePayload(AllocatedSlot.java:149)
	at org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl.tryFailingAllocatedSlot(SlotPoolImpl.java:730)
	at org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl.failAllocation(SlotPoolImpl.java:710)
	at org.apache.flink.runtime.jobmaster.JobMaster.internalFailAllocation(JobMaster.java:541)
	at org.apache.flink.runtime.jobmaster.JobMaster.notifyAllocationFailure(JobMaster.java:667)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcInvocation(AkkaRpcActor.java:274)
	at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:194)
	at org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:74)
	at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:152)
	at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26)
	at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21)
	at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123)
	at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21)
	at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:170)
	at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
	at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
	at akka.actor.Actor$class.aroundReceive(Actor.scala:517)
	at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225)
	at akka.actor.ActorCell.receiveMessage(ActorCell.scala:592)
	at akka.actor.ActorCell.invoke(ActorCell.scala:561)
	at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258)
	at akka.dispatch.Mailbox.run(Mailbox.scala:225)
	at akka.dispatch.Mailbox.exec(Mailbox.scala:235)
	at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
	at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
	at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
	at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: java.util.concurrent.TimeoutException: The heartbeat of TaskManager with id container_e14_1594608422123_4297_01_000002  timed out.
	at org.apache.flink.runtime.resourcemanager.ResourceManager$TaskManagerHeartbeatListener.notifyHeartbeatTimeout(ResourceManager.java:1125)
	at org.apache.flink.runtime.heartbeat.HeartbeatMonitorImpl.run(HeartbeatMonitorImpl.java:109)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRunAsync(AkkaRpcActor.java:397)
	at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:190)
	... 20 more
2020-08-20 09:22:52,698 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source: Custom Source -> (Sink: Print to Std. Out, Map -> Filter) (2/3) (0d127d02ced19432001f821da02cdc8c) switched from RUNNING to CANCELING.
2020-08-20 09:22:52,700 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source: Custom Source -> (Sink: Print to Std. Out, Map -> Filter) (3/3) (2486309ff7807d190e2131e2dde46a3d) switched from RUNNING to CANCELING.
2020-08-20 09:22:52,700 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Discarding the results produced by task execution cbe68593452a4ede5106d642b57c5b4d.
2020-08-20 09:22:52,702 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source: Custom Source -> (Sink: Print to Std. Out, Map -> Filter) (2/3) (0d127d02ced19432001f821da02cdc8c) switched from CANCELING to CANCELED.
2020-08-20 09:22:52,702 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Discarding the results produced by task execution 0d127d02ced19432001f821da02cdc8c.
2020-08-20 09:22:52,702 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source: Custom Source -> (Sink: Print to Std. Out, Map -> Filter) (3/3) (2486309ff7807d190e2131e2dde46a3d) switched from CANCELING to CANCELED.
2020-08-20 09:22:52,702 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Discarding the results produced by task execution 2486309ff7807d190e2131e2dde46a3d.
2020-08-20 09:22:52,702 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Job Label2ES Streaming  (3dc5e025bf4569cf32f17438317a13d1) switched from state FAILING to FAILED.
org.apache.flink.runtime.JobException: Recovery is suppressed by NoRestartBackoffTimeStrategy
	at org.apache.flink.runtime.executiongraph.failover.flip1.ExecutionFailureHandler.handleFailure(ExecutionFailureHandler.java:110)
	at org.apache.flink.runtime.executiongraph.failover.flip1.ExecutionFailureHandler.getFailureHandlingResult(ExecutionFailureHandler.java:76)
	at org.apache.flink.runtime.scheduler.DefaultScheduler.handleTaskFailure(DefaultScheduler.java:192)
	at org.apache.flink.runtime.scheduler.DefaultScheduler.maybeHandleTaskFailure(DefaultScheduler.java:186)
	at org.apache.flink.runtime.scheduler.DefaultScheduler.updateTaskExecutionStateInternal(DefaultScheduler.java:180)
	at org.apache.flink.runtime.scheduler.SchedulerBase.updateTaskExecutionState(SchedulerBase.java:484)
	at org.apache.flink.runtime.scheduler.UpdateSchedulerNgOnInternalFailuresListener.notifyTaskFailure(UpdateSchedulerNgOnInternalFailuresListener.java:49)
	at org.apache.flink.runtime.executiongraph.ExecutionGraph.notifySchedulerNgAboutInternalTaskFailure(ExecutionGraph.java:1703)
	at org.apache.flink.runtime.executiongraph.Execution.processFail(Execution.java:1252)
	at org.apache.flink.runtime.executiongraph.Execution.processFail(Execution.java:1220)
	at org.apache.flink.runtime.executiongraph.Execution.fail(Execution.java:955)
	at org.apache.flink.runtime.jobmaster.slotpool.SingleLogicalSlot.signalPayloadRelease(SingleLogicalSlot.java:173)
	at org.apache.flink.runtime.jobmaster.slotpool.SingleLogicalSlot.release(SingleLogicalSlot.java:165)
	at org.apache.flink.runtime.jobmaster.slotpool.SlotSharingManager$SingleTaskSlot.release(SlotSharingManager.java:732)
	at org.apache.flink.runtime.jobmaster.slotpool.SlotSharingManager$MultiTaskSlot.release(SlotSharingManager.java:537)
	at org.apache.flink.runtime.jobmaster.slotpool.AllocatedSlot.releasePayload(AllocatedSlot.java:149)
	at org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl.tryFailingAllocatedSlot(SlotPoolImpl.java:730)
	at org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl.failAllocation(SlotPoolImpl.java:710)
	at org.apache.flink.runtime.jobmaster.JobMaster.internalFailAllocation(JobMaster.java:541)
	at org.apache.flink.runtime.jobmaster.JobMaster.notifyAllocationFailure(JobMaster.java:667)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcInvocation(AkkaRpcActor.java:274)
	at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:194)
	at org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:74)
	at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:152)
	at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26)
	at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21)
	at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123)
	at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21)
	at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:170)
	at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
	at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
	at akka.actor.Actor$class.aroundReceive(Actor.scala:517)
	at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225)
	at akka.actor.ActorCell.receiveMessage(ActorCell.scala:592)
	at akka.actor.ActorCell.invoke(ActorCell.scala:561)
	at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258)
	at akka.dispatch.Mailbox.run(Mailbox.scala:225)
	at akka.dispatch.Mailbox.exec(Mailbox.scala:235)
	at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
	at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
	at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
	at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: java.util.concurrent.TimeoutException: The heartbeat of TaskManager with id container_e14_1594608422123_4297_01_000002  timed out.
	at org.apache.flink.runtime.resourcemanager.ResourceManager$TaskManagerHeartbeatListener.notifyHeartbeatTimeout(ResourceManager.java:1125)
	at org.apache.flink.runtime.heartbeat.HeartbeatMonitorImpl.run(HeartbeatMonitorImpl.java:109)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRunAsync(AkkaRpcActor.java:397)
	at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:190)
	... 20 more
2020-08-20 09:22:52,703 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator     - Stopping checkpoint coordinator for job 3dc5e025bf4569cf32f17438317a13d1.
2020-08-20 09:22:52,703 INFO  org.apache.flink.runtime.checkpoint.StandaloneCompletedCheckpointStore  - Shutting down
2020-08-20 09:22:52,706 INFO  org.apache.flink.runtime.dispatcher.MiniDispatcher            - Job 3dc5e025bf4569cf32f17438317a13d1 reached globally terminal state FAILED.
2020-08-20 09:22:52,707 INFO  org.apache.flink.runtime.jobmaster.JobMaster                  - Stopping the JobMaster for job Label2ES Streaming (3dc5e025bf4569cf32f17438317a13d1).
2020-08-20 09:22:52,708 INFO  org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl      - Suspending SlotPool.
2020-08-20 09:22:52,708 INFO  org.apache.flink.runtime.jobmaster.JobMaster                  - Close ResourceManager connection 185c1b05023a28dabdc753bafaeea7b2: JobManager is shutting down..
2020-08-20 09:22:52,708 INFO  org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl      - Stopping SlotPool.
2020-08-20 09:22:52,708 INFO  org.apache.flink.yarn.YarnResourceManager                     - Disconnect job manager 00000000000000000000000000000000@akka.tcp://flink@10.0.0.98:32812/user/jobmanager_0 for job 3dc5e025bf4569cf32f17438317a13d1 from the resource manager.
2020-08-20 09:22:52,709 INFO  org.apache.flink.runtime.jobmaster.JobManagerRunnerImpl       - JobManagerRunner already shutdown.
2020-08-20 09:24:17,137 WARN  akka.remote.transport.netty.NettyTransport                    - Remote connection to [/10.0.0.98:54188] failed with java.io.IOException: Connection reset by peer
2020-08-20 09:24:17,143 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink@10.0.0.98:45661] has failed, address is now gated for [50] ms. Reason: [Disassociated] 
2020-08-20 09:24:17,143 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink-metrics@10.0.0.98:35742] has failed, address is now gated for [50] ms. Reason: [Disassociated] 
2020-08-20 15:27:49,595 ERROR org.apache.flink.runtime.rest.handler.taskmanager.TaskManagerDetailsHandler  - Unhandled exception.
org.apache.flink.runtime.resourcemanager.exceptions.UnknownTaskExecutorException: No TaskExecutor registered under container_e14_1594608422123_4297_01_000002.
	at org.apache.flink.runtime.resourcemanager.ResourceManager.requestTaskManagerInfo(ResourceManager.java:532)
	at sun.reflect.GeneratedMethodAccessor40.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcInvocation(AkkaRpcActor.java:279)
	at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:194)
	at org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:74)
	at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:152)
	at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26)
	at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21)
	at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123)
	at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21)
	at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:170)
	at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
	at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
	at akka.actor.Actor$class.aroundReceive(Actor.scala:517)
	at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225)
	at akka.actor.ActorCell.receiveMessage(ActorCell.scala:592)
	at akka.actor.ActorCell.invoke(ActorCell.scala:561)
	at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258)
	at akka.dispatch.Mailbox.run(Mailbox.scala:225)
	at akka.dispatch.Mailbox.exec(Mailbox.scala:235)
	at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
	at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
	at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
	at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
2020-08-20 15:27:52,605 ERROR org.apache.flink.runtime.rest.handler.taskmanager.TaskManagerDetailsHandler  - Unhandled exception.
org.apache.flink.runtime.resourcemanager.exceptions.UnknownTaskExecutorException: No TaskExecutor registered under container_e14_1594608422123_4297_01_000002.
	at org.apache.flink.runtime.resourcemanager.ResourceManager.requestTaskManagerInfo(ResourceManager.java:532)
	at sun.reflect.GeneratedMethodAccessor40.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcInvocation(AkkaRpcActor.java:279)
	at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:194)
	at org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:74)
	at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:152)
	at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26)
	at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21)
	at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123)
	at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21)
	at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:170)
	at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
	at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
	at akka.actor.Actor$class.aroundReceive(Actor.scala:517)
	at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225)
	at akka.actor.ActorCell.receiveMessage(ActorCell.scala:592)
	at akka.actor.ActorCell.invoke(ActorCell.scala:561)
	at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258)
	at akka.dispatch.Mailbox.run(Mailbox.scala:225)
	at akka.dispatch.Mailbox.exec(Mailbox.scala:235)
	at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
	at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
	at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
	at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

日志太长直接看主要的地方吧：

2. 定位问题

 2.1 主要的问题就是超时异常，连接不上容器container 了

java.util.concurrent.TimeoutException: The heartbeat of TaskManager with id container_e14_1594608422123_4297_01_000002  timed out.

2.2 查看是否是因为资源不够，网络延迟等引起的问题，发现只有一个container，一个core，这个时候有点疑惑了为什么只有一个呢？

2.3 先理解一下flink on yarn perjob模式中的taskslots 是如何对应yarn中的container 跟core数的

 在服务器上运行，具体的参数解释可以看一下这篇博客：https://blog.csdn.net/Zsigner/article/details/107787344

flink run --help

这里主要了解 -ys ,以及 -p 参数

参数释义理解
-ysNumber of slots per TaskManager每一个taskmanager里面有多少个slots
-pThe parallelism with which to run the
program. Optional flag to override the
default value specified in the
configuration.		设置每个任务的并行度

yarn 中的core数 对应taskmanager的slots数量

yarn中的container数 = taskmanager 数量（理解是节点数）= -p / -ys

测试求证：

运行如下命令

flink run -m yarn-cluster -ynm mainname  -ys 3 -p 30 -yjm 10240 -ytm 20480  -c com.group.mainclass xxx.jar

图示如下请看：

3.解决问题

 3.1 原来的执行命令是 -ys 3 -p 3 ，只有一个taskmanager，假设这个挂了程序也就挂了

flink run -m yarn-cluster -ynm mainname  -ys 3 -p 3 -yjm 1024 -ytm 2048  -c com.group.mainclass xxx.jar

3.2 命令修改如下，（最好结合自身的业务场景设置并不是越大越好）

flink run -m yarn-cluster -ynm mainname  -ys 4 -p 20 -yjm 10240 -ytm 20480  -c com.group.mainclass xxx.jar

 这样子虽然程序已经正常跑起来了，可是还是没有解决我标题的问题，为什么在yarn中应用还是Running，在WebUI中已经是failed掉了？？？？？

 3.3 查看参数 -d

-d,--detachedIf present,runs the job in detached mode

参考文章里面说加了-d ：YarnSessionClusterEntrypoint 变成了 YarnJobClusterEntrypoint
http://apache-flink.147419.n8.nabble.com/flink1-9-1-fail-yarn-application-running-td5657.html

可是我的一开始就是YarnJobClusterEntrypoint

死马当活马医了，我先加上如果后面没有补充博客内容，就是可以了，否则.....没有否则看天意吧。

4. 最终执行命令

flink run -d -m yarn-cluster -ynm mainname  -ys 4 -p 20 -yjm 10240 -ytm 20480  -c com.group.mainclass xxx.jar

2020-08-27 15:46:05------------------------------------------------------------------------------------------------------------------------

好的，我胡汉三又回来了，上面的坑终于填上了，亲测上面的参数 -d 可行，今天程序又停了，经历了一波电话轰炸滋味挺美妙的，有点怀念程序失败了还在yarn running的时候。

Zsigner
关注
专栏目录
Flink中三种模式：YARN Session 模式、YARN Per-Job 模式和 YARN Application 模式提交任务命令
JustinMars的博客
07-30 415
在 Apache Flink 中，YARN 提供了多种模式来提交和管理作业，每种模式都有其独特的特点和适用场景。主要有以下三种模式：YARN Session 模式、YARN Per-Job 模式和 YARN Application 模式。
Apache Flink连载（二十一）：Flink On Yarn运行原理-Yarn Application模式
最新发布
qq_32020645的博客
12-30 970
🏡 个人主页：IT贫道_大数据OLAP体系技术栈,Apache Doris,Clickhouse 技术-CSDN博客🚩 私聊博主：加入大数据技术讨论群聊，获取更多大数据资料。🔔 博主个人B栈地址：豹哥教你大数据的个人空间-豹哥教你大数据个人主页-哔哩哔哩视频。
提交Flink的jar包到Yarn上，web界面log报错
piaoken5588的博客
03-14 2963
在hadoop环境下运行flink的batch程序： ./flink run /home/hadoop/proj/wordcount.jar --input hdfs://mycluster/flinkProjectData/111.txt 一段时间后，flink的web界面上，taskmanager和jobmanager下的logs分别出现error： taskmanager： ER...
yarn 命令学习：yarn application
jkdcdlly的专栏
12-05 6755
yarn application 说明：打印  application(s) 报告，或者 kill 掉 application，共7个可选项 1、-list          列出所有 application 信息     示例：yarn  application -list     打印信息： Application-Id Applicat
Yarn在运行时报错error handling URI: /cluster org.eclipse.jetty.io.RuntimeIOException: org.eclipse.jetty.io
zhang5324496的博客
08-27 3547
在Yarn运行的application的日志中发现有规律的错误日志 影响：并没有影响服务，application正常处理数据。 日志现状： 1-- 两个时间点。每个时间点往后间隔1分钟会有相同错误日志。 2-- 不是每次间隔都会有(即使1分钟的整数倍，有时候2分钟，有时候4分钟)。 日志内容： 2020-07-01 10:05:43,497 ERROR webapp.Dispatcher (Dispatcher.java:service(171)) - error handling URI:
HDP 中 hadoop yarn jobhistory 无法访问web ui 中/log 页面的问题
qq_28112019的博客
02-08 3157
HDP hadoop web 无法访问log页面
hadoop yarn web无法访问
新的开始的专栏
10-11 6895
hadoop集群正常启动后，hdfs的web可以访问，但是yarn的web页面无法访问，查看yarn的日志，也是正常启动，没有异常日志，后来以为端口的问题，查看端口的时候，发现了不同： 8088对应的local Address是docker本身的ip，和50070对应的是不一样，然后在yarn-site.xml添加配置： <!--yarn总管理器的web http通讯地址--> <property> <name>yarn.resou...
【Flink 实战系列】Flink on yarn 为什么 Allocated CPU VCores 显示不正确？
JasonLee实时计算
09-03 1049
任务的并行度也是 3，所以只需要一个 TM 即可，那么问题就来了，这个任务一共需要多少个 CPU VCores，很明显答案应该是 4 个，3（TM）+ 1 (JM) = 4 个，这里不要忘了，JM 也是需要一个 CPU 的，那我们再来看下 Yarn UI 上面显示的 CPU VCores 是多少个呢？这个问题被问到了好多次，今天就来详细的解释一下原因。但是在生产环境中可能会遇到一个问题，任务的 CPU 使用率比较低，可能会出现浪费 CPU 资源的情况，比如这个任务的 CPU 使用率。
flink实战--flink on yarn
阿华田的博客
10-31 5680
Flink on yarn 配置 如果我们打算把flink任务提交到yarn上，那么只需要一台flink客户机即可，但是需要有如下的配置 第一步：设置HADOOP_CONF_DIR环境变量 export HADOOP_CONF_DIR=/usr/local/service/hadoop/etc/hadooop/ 注意：如果在flink的配置文件中设置Hadoop的环境变量，......
flink的onYarn模式环境搭建
qw3276425405的博客
05-27 1253
一.onYarn模式 把资源管理器交给yarn模式进行管理 1.两种运行模式 1）session-cluster模式 session：会话，开辟一块新的公共的空间 先启动集群，直接向yarn申请空间，资源（session）保持不变，这个集群会常驻在yarn的集群中，不需要时进行手动停止，job任务可以向这个session提交，所有任务共享这个session，如果资源用完，需要等待之间的任务执行完毕释放资源才可以使用。 例：有100个工人，装修公司有5家，a公司有30人，b公司有10人等，a公司不
Flink on yarn部署及任务提交
qq_24095055的博客
08-04 2807
文章目录1. 完成hadoop本地配置后启动2. flink提交作业到yarn的两种方式2.1 Flink ON Yarn启动流程2.2 Flink YARN Session2.2 Run a single Flink job on YARN Hadoop单机yarn配置可参考博客，除了里面提到的./start-yarn.sh外，也可以使用./start-all.sh。 1. 完成hadoop本地配置后启动 # yarn方式启动，也可以使用 ./start-yarn.sh ~/software/hadoop
yarn app的状态 ALL, NEW, NEW_SAVING, SUBMITTED, ACCEPTED, RUNNING, FINISHED, FAILED, KILLED
Ahuuua的博客
06-11 5579
转自：yarn RMApp-状态机实现https://www.jianshu.com/p/cb98a72c94ab 状态机介绍 在RM内部维护着所有Application的状态。对于每个Application都有一个RMApp对象与之对应。在RMApp的实现类RMAppImpl中，维护着对象的基本信息，包括起始时间、名字、用户、组等信息，其中最复杂的部分莫过于其维护的状态机。 状态与转换解...
Flink的web界面无法访问
xkai957的博客
07-03 3340
Flink web界面访问
Yarn容量调度器并发度问题，解决一直卡在Running job
For_dongyang的专栏
03-04 3535
一、问题描述 Yarn默认调度器为CapacityScheduler（容量调度器），且默认只有一个队列——default。如果队列中执行第一个任务资源不够，就不会再执行第二个任务，一直等到第一个任务执行完毕。 （1）创建一张测试表 create table student(id int, name string); （2）启动1个hive客户端，执行以下插入数据的sql语句 insert into table student values(1,'abc'); 执行该语句，hive会初始化..
Hive提交到yarn的任务一直在running问题排查-V2
热门推荐
u013332124的专栏
12-24 3万+
解决hive已经停止的任务一直在yarn上运行的问题
flink中使用webUI查看信息时job not found
@coredao
04-03 1341
A1 报错 这个报错不影响执行 A2 原因 上一次任务的webUI界面没有关闭，导致flink请求一个不存在的网页 A3 解决 关闭上次任务的网页（删掉后面的url信息）
flink启动后web访问问题
lubingxue7048的专栏
11-01 5102
在执行./start-cluster.sh 后，flink进程的监听ip端口是。这样只能本机进行访问，外部服务器是访问不了的，出现这个问题的原因是flink的。所以启动后就解析到127.0.0.1了。, 进程启动时会按照这个主机名从。
flink yarn模式提交任务出错
weixin_50656879的博客
03-27 949
flink yarn方式提交出现错误的分析
Flink 读取HDFS 报错: Caused by: org.apache.flink.runtime.JobException: Recovery is suppressed by NoResta
向热爱自己一样，热爱这个世界。
03-29 1063
Flink 读取HDFS 报错: Caused by: org.apache.flink.runtime.JobException: Recovery is suppressed by NoRestartBackoffTimeStrategy
flink on yarn 多集群监控
05-28
- `<yarn-application-id>` 为 Flink on YARN 应用程序的 Application ID，可以在 YARN 的 ResourceManager Web UI 中找到。 通过 Flink Web UI，您可以查看 Flink 应用程序和任务的状态、指标、日志等信息，还可以...
评论 3
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包

打赏作者

Zsigner

你的鼓励将是我创作的最大动力

¥1 ¥2 ¥4 ¥6 ¥10 ¥20
扫码支付：¥1
获取中
扫码支付

您的余额不足，请更换扫码支付或充值

打赏作者

实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值