【性能|优化】TB级flink任务报错分析：Could not compute the container Resource

已于 2022-11-29 00:31:23 修改
阅读量1.7k 收藏 3
点赞数 1
分类专栏： # flink性能调优 # flink 实战 文章标签： flink 大数据 hadoop
于 2022-11-29 00:29:17 首次发布
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/hiliang521/article/details/128089291
版权

文章目录

一. 问题引入

1. 场景描述

使用flink引擎，处理hdfs到hive的任务，hdfs的文件数有4000个，这里设置并行度为20，提交任务运行。

 

2. 日志简析

任务提交之后发现报错，我们简单分析下yarn的日志：

1. 申请的资源超过了yarn最大的container资源限制，也就是说一个taskexecutor所需的资源过大

Could not compute the container Resource from the given TaskExecutorProcessSpec TaskExecutorProcessSpec {cpuCores=10.0, frameworkHeapSize=128.000mb (134217728 bytes), frameworkOffHeapSize=128.000mb (134217728 bytes), taskHeapSize=5.200gb (5583457416 bytes), taskOffHeapSize=0 bytes, networkMemSize=1024.000mb (1073741824 bytes), managedMemorySize=4.300gb (4617089912 bytes), jvmMetaspaceSize=256.000mb (268435456 bytes), jvmOverheadSize=1024.000mb (1073741824 bytes)}. This usually indicates the requested resource is larger than Yarn's max container resource limit.




2. 接着开始请求一个新的worker，这里的worker应该也是container ，那此时正在pend的数量为1.

Requesting new worker with resource spec WorkerResourceSpec {cpuCores=10.0, taskHeapSize=5.200gb (5583457416 bytes), taskOffHeapSize=0 bytes, networkMemSize=1024.000mb (1073741824 bytes), managedMemSize=4.300gb (4617089912 bytes)}, current pending count: 1.



3. 但同样的这个container超过了yarn的资源，这时直接放弃分配资源

Requested container resource (<memory:12288, vCores:10>) exceeds the max limitation of the Yarn cluster (<memory:12288, vCores:4>). Will not allocate resource.



4. 如此在反复这样执行，似乎陷入到了死循环。

最后任务部署超时而报错。。。

上面的日志说的比较明白，就是：申请的资源超过了yarn最大的container资源限制。
 
 
再放出点堆栈信息，供日后参考分析：

2022-11-18 <b>10:12:12</b>,938 WARN  org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager [] - Failed requesting worker with resource spec WorkerResourceSpec {cpuCores=10.0, taskHeapSize=5.200gb (5583457416 bytes), taskOffHeapSize=0 bytes, networkMemSize=1024.000mb (1073741824 bytes), managedMemSize=4.300gb (4617089912 bytes)}, current pending count: 0
org.apache.flink.runtime.resourcemanager.exceptions.ResourceManagerException: <b>Could not compute the container Resource from the given</b> TaskExecutorProcessSpec TaskExecutorProcessSpec {cpuCores=10.0, frameworkHeapSize=128.000mb (134217728 bytes), frameworkOffHeapSize=128.000mb (134217728 bytes), taskHeapSize=5.200gb (5583457416 bytes), taskOffHeapSize=0 bytes, networkMemSize=1024.000mb (1073741824 bytes), managedMemorySize=4.300gb (4617089912 bytes), jvmMetaspaceSize=256.000mb (268435456 bytes), jvmOverheadSize=1024.000mb (1073741824 bytes)}. This usually indicates the requested resource is larger than Yarn's max container resource limit.
	。。。
	at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) [flink-dist_2.12-1.12.7.jar:1.12.7]
2022-11-18 <b>10:12:12,93</b>9 INFO  org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager [] - Requesting new worker with resource spec WorkerResourceSpec {cpuCores=10.0, taskHeapSize=5.200gb (5583457416 bytes), taskOffHeapSize=0 bytes, networkMemSize=1024.000mb (1073741824 bytes), managedMemSize=4.300gb (4617089912 bytes)}, current pending count: 1.
2022-11-18 10:12:12,940 WARN  org.apache.flink.yarn.TaskExecutorProcessSpecContainerResourcePriorityAdapter [] - Requested container resource (<memory:12288, vCores:10>) exceeds the max limitation of the Yarn cluster (<memory:12288, vCores:4>). Will not allocate resource.
2022-11-18 10:12:12,940 WARN  org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager [] - Failed requesting worker with resource spec WorkerResourceSpec {cpuCores=10.0, taskHeapSize=5.200gb (5583457416 bytes), taskOffHeapSize=0 bytes, networkMemSize=1024.000mb (1073741824 bytes), managedMemSize=4.300gb (4617089912 bytes)}, current pending count: 0
org.apache.flink.runtime.resourcemanager.exceptions.ResourceManagerException: Could not compute the container Resource from the given TaskExecutorProcessSpec TaskExecutorProcessSpec {cpuCores=10.0, frameworkHeapSize=128.000mb (134217728 bytes), frameworkOffHeapSize=128.000mb (134217728 bytes), taskHeapSize=5.200gb (5583457416 bytes), taskOffHeapSize=0 bytes, networkMemSize=1024.000mb (1073741824 bytes), managedMemorySize=4.300gb (4617089912 bytes), jvmMetaspaceSize=256.000mb (268435456 bytes), jvmOverheadSize=1024.000mb (1073741824 bytes)}. This usually indicates the requested resource is larger than Yarn's max container resource limit.
	。。。
	at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) [flink-dist_2.12-1.12.7.jar:1.12.7]
2022-11-18 10:12:12,941 INFO  org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager [] - Requesting new worker with resource spec WorkerResourceSpec {cpuCores=10.0, taskHeapSize=5.200gb (5583457416 bytes), taskOffHeapSize=0 bytes, networkMemSize=1024.000mb (1073741824 bytes), managedMemSize=4.300gb (4617089912 bytes)}, current pending count: 1.
2022-11-18 10:12:12,941 WARN  org.apache.flink.yarn.TaskExecutorProcessSpecContainerResourcePriorityAdapter [] - Requested container resource (<memory:12288, vCores:10>) exceeds the max limitation of the Yarn cluster (<memory:12288, vCores:4>). Will not allocate resource.
2022-11-18 10:12:12,941 WARN  org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager [] - Failed requesting worker with resource spec WorkerResourceSpec {cpuCores=10.0, taskHeapSize=5.200gb (5583457416 bytes), taskOffHeapSize=0 bytes, networkMemSize=1024.000mb (1073741824 bytes), managedMemSize=4.300gb (4617089912 bytes)}, current pending count: 0
org.apache.flink.runtime.resourcemanager.exceptions.ResourceManagerException: Could not compute the container Resource from the given TaskExecutorProcessSpec TaskExecutorProcessSpec {cpuCores=10.0, frameworkHeapSize=128.000mb (134217728 bytes), frameworkOffHeapSize=128.000mb (134217728 bytes), taskHeapSize=5.200gb (5583457416 bytes), taskOffHeapSize=0 bytes, networkMemSize=1024.000mb (1073741824 bytes), managedMemorySize=4.300gb (4617089912 bytes), jvmMetaspaceSize=256.000mb (268435456 bytes), jvmOverheadSize=1024.000mb (1073741824 bytes)}. This usually indicates the requested resource is larger than Yarn's max container resource limit.
	at org.apache.flink.yarn.YarnResourceManagerDriver.requestResource(YarnResourceManagerDriver.java:254) ~[flink-dist_2.12-1.12.7.jar:1.12.7]
	at org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager.requestNewWorker(ActiveResourceManager.java:249) ~[flink-dist_2.12-1.12.7.jar:1.12.7]
	at org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager.requestWorkerIfRequired(ActiveResourceManager.java:310) ~[flink-dist_2.12-1.12.7.jar:1.12.7]
	at org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager.lambda$requestNewWorker$0(ActiveResourceManager.java:261) ~[flink-dist_2.12-1.12.7.jar:1.12.7]
	at java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:822) ~[?:1.8.0_152]
	at java.util.concurrent.CompletableFuture.uniHandleStage(CompletableFuture.java:834) ~[?:1.8.0_152]
	at java.util.concurrent.CompletableFuture.handle(CompletableFuture.java:2155) ~[?:1.8.0_152]
	at org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager.requestNewWorker(ActiveResourceManager.java:251) ~[flink-dist_2.12-1.12.7.jar:1.12.7]
	at org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager.requestWorkerIfRequired(ActiveResourceManager.java:310) ~[flink-dist_2.12-1.12.7.jar:1.12.7]
	at org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager.lambda$requestNewWorker$0(ActiveResourceManager.java:261) ~[flink-dist_2.12-1.12.7.jar:1.12.7]
	at java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:822) ~[?:1.8.0_152]
	at java.util.concurrent.CompletableFuture.uniHandleStage(CompletableFuture.java:834) ~[?:1.8.0_152]
	at java.util.concurrent.CompletableFuture.handle(CompletableFuture.java:2155) ~[?:1.8.0_152]
	at org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager.requestNewWorker(ActiveResourceManager.java:251) ~[flink-dist_2.12-1.12.7.jar:1.12.7]
	at org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager.requestWorkerIfRequired(ActiveResourceManager.java:310) ~[flink-dist_2.12-1.12.7.jar:1.12.7]
	at org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager.lambda$requestNewWorker$0(ActiveResourceManager.java:261) ~[flink-dist_2.12-1.12.7.jar:1.12.7]
	at java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:822) ~[?:1.8.0_152]
	at java.util.concurrent.CompletableFuture.uniHandleStage(CompletableFuture.java:834) ~[?:1.8.0_152]
	at java.util.concurrent.CompletableFuture.handle(CompletableFuture.java:2155) ~[?:1.8.0_152]
	at org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager.requestNewWorker(ActiveResourceManager.java:251) ~[flink-dist_2.12-1.12.7.jar:1.12.7]
	at org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager.requestWorkerIfRequired(ActiveResourceManager.java:310) ~[flink-dist_2.12-1.12.7.jar:1.12.7]
	at org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager.lambda$requestNewWorker$0(ActiveResourceManager.java:261) ~[flink-dist_2.12-1.12.7.jar:1.12.7]
	at java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:822) ~[?:1.8.0_152]
	at java.util.concurrent.CompletableFuture.uniHandleStage(CompletableFuture.java:834) ~[?:1.8.0_152]
	at java.util.concurrent.CompletableFuture.handle(CompletableFuture.java:2155) ~[?:1.8.0_152]
	at org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager.requestNewWorker(ActiveResourceManager.java:251) ~[flink-dist_2.12-1.12.7.jar:1.12.7]
	at org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager.requestWorkerIfRequired(ActiveResourceManager.java:310) ~[flink-dist_2.12-1.12.7.jar:1.12.7]
	at org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager.lambda$requestNewWorker$0(ActiveResourceManager.java:261) ~[flink-dist_2.12-1.12.7.jar:1.12.7]
	at java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:822) ~[?:1.8.0_152]
	at java.util.concurrent.CompletableFuture.uniHandleStage(CompletableFuture.java:834) ~[?:1.8.0_152]
	at java.util.concurrent.CompletableFuture.handle(CompletableFuture.java:2155) ~[?:1.8.0_152]
	at org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager.requestNewWorker(ActiveResourceManager.java:251) ~[flink-dist_2.12-1.12.7.jar:1.12.7]
	at org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager.requestWorkerIfRequired(ActiveResourceManager.java:310) ~[flink-dist_2.12-1.12.7.jar:1.12.7]
	at org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager.lambda$requestNewWorker$0(ActiveResourceManager.java:261) ~[flink-dist_2.12-1.12.7.jar:1.12.7]
	at java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:822) ~[?:1.8.0_152]
	at java.util.concurrent.CompletableFuture.uniHandleStage(CompletableFuture.java:834) ~[?:1.8.0_152]
	at java.util.concurrent.CompletableFuture.handle(CompletableFuture.java:2155) ~[?:1.8.0_152]
	at org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager.requestNewWorker(ActiveResourceManager.java:251) ~[flink-dist_2.12-1.12.7.jar:1.12.7]
	at org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager.requestWorkerIfRequired(ActiveResourceManager.java:310) ~[flink-dist_2.12-1.12.7.jar:1.12.7]
	at org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager.lambda$requestNewWorker$0(ActiveResourceManager.java:261) ~[flink-dist_2.12-1.12.7.jar:1.12.7]
	at java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:822) ~[?:1.8.0_152]
	at java.util.concurrent.CompletableFuture.uniHandleStage(CompletableFuture.java:834) ~[?:1.8.0_152]
	at java.util.concurrent.CompletableFuture.handle(CompletableFuture.java:2155) ~[?:1.8.0_152]
	at org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager.requestNewWorker(ActiveResourceManager.java:251) ~[flink-dist_2.12-1.12.7.jar:1.12.7]
	at org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager.requestWorkerIfRequired(ActiveResourceManager.java:310) ~[flink-dist_2.12-1.12.7.jar:1.12.7]
	at org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager.lambda$requestNewWorker$0(ActiveResourceManager.java:261) ~[flink-dist_2.12-1.12.7.jar:1.12.7]
	at java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:822) ~[?:1.8.0_152]
	at java.util.concurrent.CompletableFuture.uniHandleStage(CompletableFuture.java:834) ~[?:1.8.0_152]
	at java.util.concurrent.CompletableFuture.handle(CompletableFuture.java:2155) ~[?:1.8.0_152]
	at org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager.requestNewWorker(ActiveResourceManager.java:251) ~[flink-dist_2.12-1.12.7.jar:1.12.7]
	at org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager.requestWorkerIfRequired(ActiveResourceManager.java:310) ~[flink-dist_2.12-1.12.7.jar:1.12.7]
	at org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager.lambda$requestNewWorker$0(ActiveResourceManager.java:261) ~[flink-dist_2.12-1.12.7.jar:1.12.7]
	at java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:822) ~[?:1.8.0_152]
	at java.util.concurrent.CompletableFuture.uniHandleStage(CompletableFuture.java:834) ~[?:1.8.0_152]
	at java.util.concurrent.CompletableFuture.handle(CompletableFuture.java:2155) ~[?:1.8.0_152]
	at org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager.requestNewWorker(ActiveResourceManager.java:251) ~[flink-dist_2.12-1.12.7.jar:1.12.7]
	at org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager.requestWorkerIfRequired(ActiveResourceManager.java:310) ~[flink-dist_2.12-1.12.7.jar:1.12.7]
	at org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager.lambda$requestNewWorker$0(ActiveResourceManager.java:261) ~[flink-dist_2.12-1.12.7.jar:1.12.7]
	at java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:822) ~[?:1.8.0_152]
	at java.util.concurrent.CompletableFuture.uniHandleStage(CompletableFuture.java:834) ~[?:1.8.0_152]
	at java.util.concurrent.CompletableFuture.handle(CompletableFuture.java:2155) ~[?:1.8.0_152]
	at org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager.requestNewWorker(ActiveResourceManager.java:251) ~[flink-dist_2.12-1.12.7.jar:1.12.7]
	at org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager.requestWorkerIfRequired(ActiveResourceManager.java:310) ~[flink-dist_2.12-1.12.7.jar:1.12.7]
	at org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager.lambda$requestNewWorker$0(ActiveResourceManager.java:261) ~[flink-dist_2.12-1.12.7.jar:1.12.7]
	at java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:822) ~[?:1.8.0_152]
	at java.util.concurrent.CompletableFuture.uniHandleStage(CompletableFuture.java:834) ~[?:1.8.0_152]
	at java.util.concurrent.CompletableFuture.handle(CompletableFuture.java:2155) ~[?:1.8.0_152]
	at org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager.requestNewWorker(ActiveResourceManager.java:251) ~[flink-dist_2.12-1.12.7.jar:1.12.7]
	at org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager.startNewWorker(ActiveResourceManager.java:160) ~[flink-dist_2.12-1.12.7.jar:1.12.7]
	at org.apache.flink.runtime.resourcemanager.ResourceManager$ResourceActionsImpl.allocateResource(ResourceManager.java:1382) ~[flink-dist_2.12-1.12.7.jar:1.12.7]
	at org.apache.flink.runtime.resourcemanager.slotmanager.SlotManagerImpl.allocateResource(SlotManagerImpl.java:1058) ~[flink-dist_2.12-1.12.7.jar:1.12.7]
	at org.apache.flink.runtime.resourcemanager.slotmanager.SlotManagerImpl.fulfillPendingSlotRequestWithPendingTaskManagerSlot(SlotManagerImpl.java:954) ~[flink-dist_2.12-1.12.7.jar:1.12.7]
	at org.apache.flink.runtime.resourcemanager.slotmanager.SlotManagerImpl.lambda$internalRequestSlot$9(SlotManagerImpl.java:943) ~[flink-dist_2.12-1.12.7.jar:1.12.7]
	at org.apache.flink.util.OptionalConsumer.ifNotPresent(OptionalConsumer.java:51) ~[flink-dist_2.12-1.12.7.jar:1.12.7]
	at org.apache.flink.runtime.resourcemanager.slotmanager.SlotManagerImpl.internalRequestSlot(SlotManagerImpl.java:941) ~[flink-dist_2.12-1.12.7.jar:1.12.7]
	at org.apache.flink.runtime.resourcemanager.slotmanager.SlotManagerImpl.registerSlotRequest(SlotManagerImpl.java:410) ~[flink-dist_2.12-1.12.7.jar:1.12.7]
	at org.apache.flink.runtime.resourcemanager.ResourceManager.requestSlot(ResourceManager.java:529) ~[flink-dist_2.12-1.12.7.jar:1.12.7]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_152]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_152]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_152]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_152]
	at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcInvocation(AkkaRpcActor.java:305) ~[flink-dist_2.12-1.12.7.jar:1.12.7]
	at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:212) ~[flink-dist_2.12-1.12.7.jar:1.12.7]
	at org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:77) ~[flink-dist_2.12-1.12.7.jar:1.12.7]
	at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:158) ~[flink-dist_2.12-1.12.7.jar:1.12.7]
	at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26) [flink-dist_2.12-1.12.7.jar:1.12.7]
	at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21) [flink-dist_2.12-1.12.7.jar:1.12.7]
	at scala.PartialFunction.applyOrElse(PartialFunction.scala:123) [flink-dist_2.12-1.12.7.jar:1.12.7]
	at scala.PartialFunction.applyOrElse$(PartialFunction.scala:122) [flink-dist_2.12-1.12.7.jar:1.12.7]
	at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21) [flink-dist_2.12-1.12.7.jar:1.12.7]
	at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171) [flink-dist_2.12-1.12.7.jar:1.12.7]
	at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172) [flink-dist_2.12-1.12.7.jar:1.12.7]
	at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172) [flink-dist_2.12-1.12.7.jar:1.12.7]
	at akka.actor.Actor.aroundReceive(Actor.scala:517) [flink-dist_2.12-1.12.7.jar:1.12.7]
	at akka.actor.Actor.aroundReceive$(Actor.scala:515) [flink-dist_2.12-1.12.7.jar:1.12.7]
	at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225) [flink-dist_2.12-1.12.7.jar:1.12.7]
	at akka.actor.ActorCell.receiveMessage(ActorCell.scala:592) [flink-dist_2.12-1.12.7.jar:1.12.7]
	at akka.actor.ActorCell.invoke(ActorCell.scala:561) [flink-dist_2.12-1.12.7.jar:1.12.7]
	at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258) [flink-dist_2.12-1.12.7.jar:1.12.7]
	at akka.dispatch.Mailbox.run(Mailbox.scala:225) [flink-dist_2.12-1.12.7.jar:1.12.7]
	at akka.dispatch.Mailbox.exec(Mailbox.scala:235) [flink-dist_2.12-1.12.7.jar:1.12.7]
	at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) [flink-dist_2.12-1.12.7.jar:1.12.7]
	at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) [flink-dist_2.12-1.12.7.jar:1.12.7]
	at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) [flink-dist_2.12-1.12.7.jar:1.12.7]
	at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) [flink-dist_2.12-1.12.7.jar:1.12.7]
<br/>

 
 

二. 初级问题分析与解决

1. 问题分析

报错很明确，也就是我们向yarn多申请了资源。所以我们关注两点：yarn的调度器设置（规定了队列，每个任务申请的限制）、程序中是如何设置的并行度的。

1.1. yarn的调度器设置

打开yarn 看到Maximum Allocation限制为<memory:12288, vCores:4> ，具体的说，就是我们申请一个container最大内存能申请12G、最大核心数为4
在这里插入图片描述
 

1.2. 程序设置

看下运行的shell的伪代码

tmp_value=`echo $parallelism 10 | awk '{if($1 > $2) print 1; else print 0;}'`
if [ $tmp_value -eq 1 ] ;then
  tm=12288
  vcores=10
  numberOfTaskSlots=10
fi
...

"yarn.containers.vcores":${vcores}
"taskmanager.numberOfTaskSlots":${numberOfTaskSlots}
...

这里可以看到冲突：当我们的并行度设置超过10时，vcores设置为10，但yarn最大让设置为4，所以会报错。。。

 

2. 问题解决

解决问题的方式也比较简单，让vcores最大保持为4，然后再运行。

这里我们设置整个job的并行度还是为20，然后申请一个container中vcores=4，那么按照一个并行度分配一个线程的逻辑

即一个taskmanager（container）中有4核对应有4个slot，那将会有20/4=5个taskmanager

 
 

三. （性能）新的问题

1. 问题描述

任务是跑起来了，但是出现了一个性能的问题：

先看下job的运行情况：

job的消费速度：

  • 20并行度下，每个taskmanager的内存为12G，每秒消费11.28万条数据，那每分钟处理的速度是676.89万条/min，每小时4.06亿条/小时

  • 2032.766839793899649 GB 每小时平均

  • 5793.385493412613869 GB 累计2.85小时
     

最终的结果

  • 53.14亿条
  • 数据量有：3.03TB
  • 消费每条数据的平均大小是：626byte。
  • 1G=1,073,741,824 bytes

最终整个job运行完，花了11个小时：
在这里插入图片描述
客户的要求是1小时内完成，但速度太慢了。。。
 

2. 理想化的最优方案

客户集群的资源是够的，所以我们不考虑资源问题，那既然这样，因为hdfs的文件总共有4000个，再有yarn最大资源分配是4（core/container），所以我们部署一个4000并行度的任务，它将运行的最快！！！

所以我将并行度设置为4000时将会有1000个taskmanager启动。等一下。1000个？？？我心疼jobmanager一秒钟先。

果然，任务还没调度完，就失败了。。。

所以理想有些简单，也没有银弹。

 

3. "PlanB"的解决方案

既然一个jobmanager不能管理这么多的taskmanager，那就降低taskmanager的数量。
。。。经过多次尝试之后，这里最终给出了方案B的设置：

并行度设置为500，taskmanager启动了125个。

任务最终处理时间缩短到：23分钟

*********************************************
nErrors              |  0
nullErrors           |  0
duplicateErrors      |  0
conversionErrors     |  0
otherErrors          |  0
numWrite             |  3164002700
byteWrite            |  17123302298224
numRead              |  3164002700
writeDuration        |  545313434
byteRead             |  1983716362408
readDuration         |  539805057
snapshotWrite        |  6328005400
*********************************************

2022-11-21 11:48:56 Start to stop flink job(3734d90205a13eff7cfa61b5c0c9686b)
2022-11-21 11:50:11 Success to stop flink job(3734d90205a13eff7cfa61b5c0c9686b)
2022-11-21 11:50:11 Flink process exit code is :0

 
 

四. 反思与迭代

针对上述的处理过程，发现几点值得思考和迭代：

  1. 我们程序和yarn进行定时通讯，及时获取yarn中调度器的设置，然后动态设置最大核心数，充分利用资源的同时，也保证了程序的稳定；
  1. 我们在设置并行度时，需要考虑jobmanager能够协调的taskmanager的数量，不是靠尝试。
     
    本文学到的经验是，设置125个taskmanager时，jobmanager是可以顶住压力的，但接下来可以分析分析flink的通讯相关的源码，以便能极大的发挥集群资源。
     
    其次对于jobmanager管理这么多节点是否可以设置flink的高可用，增加job运行的稳定度

还有一个小细节
在这里插入图片描述

  1. taskmanager的内存使用有些浪费，如上图，并未充分利用内存资源，这点我们也需思考要如何优化。

仔细点我们也可以发现

  1. 当container设置的核心数最大为4时，numberOfTaskSlots的设置其实无效了。
roman_日积跬步-终至千里
关注
  • 1
    点赞
  • 踩
  • 3
    收藏
    觉得还不错? 一键收藏
  • 打赏
    打赏
  • 2
    评论
专栏目录
Flink源码系列(YarnTaskExecutorRunner启动)-第九期
Airy的博客
05-09 1409
上一期指路 接着上一期讲到JobMaster的启动，其中又启动了slotPool，如下： 那么接下来slotPool寻回rm并向其请求slot。 1.StandaloneLeaderRetrievalService#start->JobMaster的内部类ResourceManagerLeaderListener#notifyLeaderAddress->JobMaster#notifyOfNewResourceManagerLeader private void notif
Flink in yarn 提交任务报错 created一直等待 This usually indicates the requested resource is larger than Yar
qq_45472675的博客
01-05 659
Flink on yarn 用 yarn-session.sh 启动之后 向flink提交任务时 taskManger一直在created 等待超时之后报错 This usually indicates the requested resource is larger than Yarn's max container resource limit.。注意 这块最小内存为1600MB 改小了启动会报错。这块是我改过的 本来配置4090 超过虚拟机内存。我是用虚拟机启动了三台 所以查看内存分配。
Flink源码篇 No.7-任务提交之JobManager申请资源（per-job on yarn）
pezynd的专栏
10-07 1151
第1章 启动JobManager 我们从jobManager启动的方法开始看 org.apache.flink.runtime.jobmaster.JobMaster#startJobExecution private Acknowledge startJobExecution(JobMasterId newJobMasterId) throws Exception { // 验证是否在主线程 validateRunsInMainThread(); checkNotNull(newJobM
Flink1.12源码解读——物理执行图（一）之TM实例构建过程
ws0owws0ow的博客
03-04 923
Apache Flink作为国内最火的大数据计算引擎之一，自身支持高吞吐，低延迟，exactly-once语义，有状态流等特性，阅读源码有助加深对框架的理解和认知。 Flink分为四种执行图，本章主要解析物理执行图的生成计划前半部分 本章及后续源码解读环境以生产为主：运行模式：OnYarn，HA模式：ZK，Mode：Streaming，关键逻辑解释我会备注到代码上（灰色字体）勿忽视。 此章节涉及的组件和框架的代码依赖部分大数据生态包括Yarn，Akka，ZK等。建议有此基础上阅读会提高可阅读性和实现理
Flink源码篇 No.6-任务提交之启动ResourceManager（per-job on yarn）
pezynd的专栏
10-07 733
第1章create创建并启动resourceManager 回到最初创建resourceManager的地方： org.apache.flink.runtime.entrypoint.component.DefaultDispatcherResourceManagerComponentFactory#create @Override public DispatcherResourceManagerComponent create( Configuration configuration, ..
Flink原理、实战与性能优化-高级进阶培训（小白请勿下载）
06-07
本高级进阶培训课程将深入讲解Flink的核心原理、实战应用和性能优化策略。 首先，课程会探讨Flink的核心设计思想，包括Dataflow计算模型与MapReduce模型的比较，以及Flink如何采用Chandy-Lamport分布式容错方案来...
P7级Flink大数据精华实战 从Flink大规模集群部署到Flink性能优化 Flink超级实战课程
05-19
P7级Flink大数据精华实战课程分为四个部分依次进阶的进行教学，从初识Flink及计算框架教学，到全面的FlinkOnYarn集群提交与部署，随后转而深入的进行Flink常用API教学。在后半程的课程中，讲师又对Flink中的...
高性能FlinkSQL优化技巧
03-01
MicroBatch在元素序列化效率、反压表现、吞吐和延迟性能上都要优于MiniBatch。微批处理通过增加延迟换取高吞吐，如果您有超低延迟的要求，不建议开启微批处理。通常对于聚合的场景，微批处理可以显
Flink原理、实战与性能优化.pdf
11-10
Flink原理、实战与性能优化.pdf
Flink实战：用户行为分析之热门商品TopN统计
01-20
在本篇《Flink实战：用户行为分析之热门商品TopN统计》中，我们将探讨如何利用Apache Flink处理实时用户行为数据，特别是针对热门商品的TopN统计。环境配置为Ubuntu 14、Flink 1.7.2、Scala 2.11、Kafka 2.3.0、JDK ...
docker 部署tb Error opening log file ‘/var/log/thingsboard/tb-monolith1-gc.log
qq_38377190的博客
05-29 171
注意，出现这个报错问题，说明你已经在没有给配置文件权限的情况进行了一次启动，这个时候挂载文件其实已经生成了，即使再对配置文件的权限进行赋权，仍然还是会出现上面的报错。所以想一次性解决上面的问题，需要赋权两个地方。docker 部署tb Error opening log file '/var/log/thingsboard/tb-monolith1-gc.log。2.启动过一次，已经被挂载出来的log文件，进行 chmod 777 赋权。1.配置文件底下的 log文件夹。然后重启tb，成功解决问题。
彻底搞明白YARN资源分配
m0_46168848的博客
07-28 1366
Container是以什么形式运行的？是单独的JVM进程吗？是的，每一个Container就是一个单独的JVM进程。YARN的vcore和本机的CPU核数关系？没关系。默认都是手动在yarn-default.xml中配置的，默认每个NodeManager是8个vcore，所有的NodeManager上的vcore加在一起就是整个YARN所有的vcore。每个Container能够使用的物理内存和虚拟内存是多少？
Spark/Spark-JobServer 执行SQL时任务超时
dymkkj的专栏
03-18 2452
1 问题，SQL执行时超时 2022/03/18 17:36:47 HttpRequest GET http://10.1.170.115:8080/v1/statement/20220318_093646_03065_awf2q/2 map[errorClass:java.util.concurrent.TimeoutException message:Futures timed out after [600 seconds] stack:java.util.concurrent.TimeoutExcep
Fink连接Kafka 连接不上 : Recovery is suppressed by NoRestartBackoffTimeStrategy
wuhahaq的博客
03-17 995
出错原因：Recovery is suppressed by NoRestartBackoffTimeStrategy。kafka进程掉了 flink链接不上kafka。
Flink报错之设备上没有空间
weixin_41465010的博客
12-26 788
Flink1.11.2 设备上没有空间 1.报错信息 2020-12-26 09:42:02 org.apache.flink.runtime.JobException: Recovery is suppressed by NoRestartBackoffTimeStrategy at org.apache.flink.runtime.executiongraph.failover.flip1.ExecutionFailureHandler.handleFailure(ExecutionFailureHa
flink并行度设置问题
L13763338360的博客
02-04 2876
之前写过一篇文章，介绍flink的并行度问题：https://blog.csdn.net/L13763338360/article/details/106632612 并行度的设置有几种，按优先级先后依次是： 算子级别 执行环境级别 命令行级别 配置文件级别 公司用的flink是基于开源改造的，跟开源还是有些区别，使用过程中也碰到一些问题，这里简单总结下。 有两个跟并行度相关的配置 taskmanager.numberOfTaskManagers：taskManager数量 taskmana
Flink1.12整合Hadoop3.x报错
qq_29860591的博客
11-18 2683
Flink1.12整合Hadoop3.x报错 由于项目需要，需要搞一段时间的规则预警，于是先在项目的阿里云服务器上安装了flink集群和hadoop，但是在跑官网示例的时候，直接报错了，具体信息如下： Caused by: java.lang.RuntimeException: org.apache.flink.runtime.client.JobInitializationException:...
Yarn参数配置Container优化
weixin_44059731的博客
12-17 2087
container: 内存 和 cpu的虚拟概念 memory:8192, vCores:8 资源例如120G，20%预留给系统 内存: yarn.nodemanager.resource.memory-mb 96G (CDH配置注意这个参数是在编辑每一台参数96/6=16G） yarn.scheduler.minimum-allocation-mb 1G 极限情况下 96个container yarn.scheduler.maximum-allocation-mb 96G 极限情况下 1个con
Yarn 3.x版本容器内存控制功能失效排查
热门推荐
weixin_42473019的博客
11-28 2万+
问题背景 Yarn集群中一部分节点的内存被打爆，排查发现作业使用内存超出了NodeManager的最大内存限制 找出故障时间点运行的作业，发现作业内存设置不合理，用户只设置了 mapreduce.map.java.opts=-Xmx40240m; mapreduce.reduce.java.opts=-Xmx80240m; 而没有设置 mapreduce.map.memory.mb（默认为1536） mapreduce.reduce.memory.mb（默认为3072） mapreduce.map.me
安装python flink时候报错ModuleNotFoundError: No module named 'distutils'
最新发布
02-22
安装Python Flink时报错"ModuleNotFoundError: No module named 'disutils'"是因为缺少distutils模块。distutils是Python的标准库之一，用于构建和安装Python模块。 要解决这个问题，你可以按照以下步骤进行操作： 1. 确保你已经正确安装了Python。你可以在命令行中输入`python --version`来检查Python的版本。 2. 如果你使用的是Python 2.x版本，那么distutils应该已经包含在Python的标准库中，无需额外安装。 3. 如果你使用的是Python 3.x版本，并且在安装Python时选择了"Minimize disk space usage"选项，那么distutils模块可能没有被安装。你可以尝试重新安装Python，并选择"Full installation"选项来确保distutils被正确安装。 4. 如果以上方法都无效，你可以尝试手动安装distutils模块。首先，下载distutils的安装包（通常是一个压缩文件），然后解压缩并进入解压后的目录。在命令行中运行以下命令进行安装： ``` python setup.py install ``` 这将会安装distutils模块到你的Python环境中。 如果你还有其他问题，请继续提问。

“相关推荐”对你有帮助么？

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论 2
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包

打赏作者

roman_日积跬步-终至千里

你的鼓励将是我创作的最大动力

¥1 ¥2 ¥4 ¥6 ¥10 ¥20
扫码支付：¥1
获取中
扫码支付

您的余额不足，请更换扫码支付或充值

打赏作者

实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值