之前写过一篇文章,介绍flink的并行度问题:https://blog.csdn.net/L13763338360/article/details/106632612
并行度的设置有几种,按优先级先后依次是:
- 算子级别
- 执行环境级别
- 命令行级别
- 配置文件级别
公司用的flink是基于开源改造的,跟开源还是有些区别,使用过程中也碰到一些问题,这里简单总结下。
有两个跟并行度相关的配置
- taskmanager.numberOfTaskManagers:taskManager数量
- taskmanager.numberOfTaskSlots:每个taskManager的slot数量
任务启动的时候,slot数量=numberOfTaskManagers*numberOfTaskSlots。
有两个跟任务相关的资源
- cpu核心数
- 单核内存
前提条件:申请了cpu核心数为4,内存8g,numberOfTaskManagers为4,numberOfTaskSlots为2,理论上slot数量为:4*2=8.
一般认为:4个cpu核心数和8G内存,4个taskmanager,8个slot。任务起来的时候,有多少slot就申请多少,这里应该申请8个slot,没用的闲置着,如果任务需要的slot数量超过了8,资源申请不下来任务启动失败。
实际情况
- 情况1:没有设置任何并行度,实际申请和使用1核,2g内存,1个taskmanager,2个slot。
- 情况2:设置了算子级别的并行度,有几个算子并行度均为2,实际申请和使用3核,6g内存,3个taskmanager,6个slot。
- 情况3:设置了算式级别并行度,读两个kafka集群,每个集群1个topic有5个分区,所以设置两个source的并行度均是5,还有其他算子的并行度,需要29个slot,但只分配了15个,slot不够,申请资源失败,报错如下
org.apache.flink.runtime.executiongraph.ExecutionGraph - oceanus#trace#Job livelink-data-flow (000000000000eab20000000000000009) switched from state RUNNING to FAILING.
org.apache.flink.runtime.jobmanager.scheduler.NoResourceAvailableException: Could not allocate all requires slots within timeout of 300000 ms. Slots required: 29, slots allocated: 15, previous allocation IDs: [], execution status: completed: Attempt #0 (Source: kafka-reader-_0 -> Map -> Filter (1/1)) @ org.apache.flink.runtime.jobmaster.slotpool.SingleLogicalSlot@75b2a848 - [SCHEDULED], completed: Attempt #0 (Source: kafka-reader-_1 (1/1)) @ org.apache.flink.runtime.jobmaster.slotpool.SingleLogicalSlot@7cefd6df - [SCHEDULED], completed: Attempt #0 (Source: Custom Source (1/1)) @ org.apache.flink.runtime.jobmaster.slotpool.SingleLogicalSlot@7d4da7f1 - [SCHEDULED], completed: Attempt #0 (Map (1/8)) @ org.apache.flink.runtime.jobmaster.slotpool.SingleLogicalSlot@6ceb85dd - [SCHEDULED], incomplete: java.util.concurrent.CompletableFuture@c7b90e0[Not completed, 1 dependents], completed: Attempt #0 (Map (3/8)) @ org.apache.flink.runtime.jobmaster.slotpool.SingleLogicalSlot@ea0e69e - [SCHEDULED], completed: Attempt #0 (Map (4/8)) @ org.apache.flink.runtime.jobmaster.slotpool.SingleLogicalSlot@1ec2aa94 - [SCHEDULED], completed: Attempt #0 (Map (5/8)) @ org.apache.flink.runtime.jobmaster.slotpool.SingleLogicalSlot@2e3711fc - [SCHEDULED], completed: Attempt #0 (Map (6/8)) @ org.apache.flink.runtime.jobmaster.slotpool.SingleLogicalSlot@7e1b684a - [SCHEDULED], incomplete: java.util.concurrent.CompletableFuture@7706e501[Not completed, 1 dependents], completed: Attempt #0 (Map (8/8)) @ org.apache.flink.runtime.jobmaster.slotpool.SingleLogicalSlot@7c1b677f - [SCHEDULED], incomplete: java.util.concurrent.CompletableFuture@6ab521f1[Not completed, 1 dependents], incomplete: java.util.concurrent.CompletableFuture@4f494dd5[Not completed, 1 dependents], completed: Attempt #0 (Co-Process-Broadcast (2/8)) @ org.apache.flink.runtime.jobmaster.slotpool.SingleLogicalSlot@fa797d0 - [SCHEDULED], completed: Attempt #0 (Co-Process-Broadcast (3/8)) @ org.apache.flink.runtime.jobmaster.slotpool.SingleLogicalSlot@282029c4 - [SCHEDULED], completed exceptionally: java.util.concurrent.CompletionException: java.util.concurrent.CompletionException: java.util.concurrent.TimeoutException/java.util.concurrent.CompletableFuture@6b592f98[Completed exceptionally], completed: Attempt #0 (Co-Process-Broadcast (5/8)) @ org.apache.flink.runtime.jobmaster.slotpool.SingleLogicalSlot@6efbe0c3 - [SCHEDULED], completed: Attempt #0 (Co-Process-Broadcast (6/8)) @ org.apache.flink.runtime.jobmaster.slotpool.SingleLogicalSlot@5dd1f3db - [SCHEDULED], completed: Attempt #0 (Co-Process-Broadcast (7/8)) @ org.apache.flink.runtime.jobmaster.slotpool.SingleLogicalSlot@74168fac - [SCHEDULED], completed: Attempt #0 (Co-Process-Broadcast (8/8)) @ org.apache.flink.runtime.jobmaster.slotpool.SingleLogicalSlot@56914b59 - [SCHEDULED], incomplete: java.util.concurrent.CompletableFuture@675d54b0[Not completed, 1 dependents], incomplete: java.util.concurrent.CompletableFuture@35402cab[Not completed, 1 dependents], incomplete: java.util.concurrent.CompletableFuture@75ebbe40[Not completed, 1 dependents], incomplete: java.util.concurrent.CompletableFuture@297cf827[Not completed, 1 dependents], incomplete: java.util.concurrent.CompletableFuture@2489d16c[Not completed, 1 dependents], incomplete: java.util.concurrent.CompletableFuture@10188422[Not completed, 1 dependents], incomplete: java.util.concurrent.CompletableFuture@382931df[Not completed, 1 dependents], incomplete: java.util.concurrent.CompletableFuture@765a5b3d[Not completed, 1 dependents], incomplete: java.util.concurrent.CompletableFuture@1c32e62b[Not completed, 1 dependents]
at org.apache.flink.runtime.executiongraph.SchedulingUtils.lambda$scheduleEager$1(SchedulingUtils.java:194)
at java.util.concurrent.CompletableFuture.uniExceptionally(CompletableFuture.java:884)
at java.util.concurrent.CompletableFuture$UniExceptionally.tryFire(CompletableFuture.java:866)
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990)
at org.apache.flink.runtime.concurrent.FutureUtils$ResultConjunctFuture.handleCompletedFuture(FutureUtils.java:634)
at org.apache.flink.runtime.concurrent.FutureUtils$ResultConjunctFuture.lambda$new$0(FutureUtils.java:657)
at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990)
at org.apache.flink.runtime.jobmaster.slotpool.SchedulerImpl.lambda$internalAllocateSlot$0(SchedulerImpl.java:190)
at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990)
at org.apache.flink.runtime.jobmaster.slotpool.SlotSharingManager$SingleTaskSlot.release(SlotSharingManager.java:700)
at org.apache.flink.runtime.jobmaster.slotpool.SlotSharingManager$MultiTaskSlot.release(SlotSharingManager.java:484)
at org.apache.flink.runtime.jobmaster.slotpool.SlotSharingManager$MultiTaskSlot.lambda$new$0(SlotSharingManager.java:380)
at java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:836)
at java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:811)
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990)
at org.apache.flink.runtime.concurrent.FutureUtils$Timeout.run(FutureUtils.java:999)
at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRunAsync(AkkaRpcActor.java:397)
at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:190)
at org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:74)
at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:152)
at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26)
at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21)
at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123)
at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21)
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:170)
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
at akka.actor.Actor$class.aroundReceive(Actor.scala:517)
at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:592)
at akka.actor.ActorCell.invoke(ActorCell.scala:561)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258)
at akka.dispatch.Mailbox.run(Mailbox.scala:225)
at akka.dispatch.Mailbox.exec(Mailbox.scala:235)
at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
为啥已经分配了15个slot,不是最多8个?
看来并行度、numberOfTaskManagers、numberOfTaskSlots、cpu、内存、实际slot数量之间的关系,有一定迷惑性。
- 并行度不好设置,并行度设置小了,资源没有充分利用;并行度大了,资源申请不了,任务起不来。
- cpu、内存申请和numberOfTaskManagers、numberOfTaskSlots配置也有一定关系,需要注意。
前面说了,设置并行度的方法,按优先级先后:算子级别、执行环境级别、命令行级别、配置文件级别,既然算子级别设置并行度,不好控制实际启动的taskmanager和slot的数量,那就试试其他的。
env.setParallelism(8);
执行环境级别,设置全局并行度为8,结果如预期:4核、8G、4个taskmanager、8个slot,总共5个task,比之前自己在算子级别设置并行度更少task。
为什么会这样?需要重新研究下parallism、subtask、task、slot、operator chain等概念,另外还需要研究下sharing group slot概念,内容比较多,这里不详细介绍。