Flink1.11内存模型与参数调整

Flink TaskManager启动日志

INFO  [] - Loading configuration property: taskmanager.memory.process.size, 2048m
INFO  [] - Loading configuration property: taskmanager.memory.managed.fraction, 0.3
INFO  [] - Loading configuration property: taskmanager.memory.jvm-overhead.fraction, 0.2
INFO  [] - Loading configuration property: taskmanager.memory.network.fraction, 0.1
INFO  [] - Loading configuration property: parallelism.default, 2
INFO  [] - Loading configuration property: task.cancellation.timeout, 0
INFO  [] - Final TaskExecutor Memory configuration:
INFO  [] -   Total Process Memory:          2.000gb (2147483648 bytes)
INFO  [] -     Total Flink Memory:          1.350gb (1449551456 bytes)
INFO  [] -       Total JVM Heap Memory:     701.440mb (735513127 bytes)
INFO  [] -         Framework:               128.000mb (134217728 bytes)
INFO  [] -         Task:                    573.440mb (601295399 bytes)
INFO  [] -       Total Off-heap Memory:     680.960mb (714038329 bytes)
INFO  [] -         Managed:                 414.720mb (434865454 bytes)
INFO  [] -         Total JVM Direct Memory: 266.240mb (279172875 bytes)
INFO  [] -           Framework:             128.000mb (134217728 bytes)
INFO  [] -           Task:                  0 bytes
INFO  [] -           Network:               138.240mb (144955147 bytes)
INFO  [] -     JVM Metaspace:               256.000mb (268435456 bytes)
INFO  [] -     JVM Overhead:                409.600mb (429496736 bytes)

1. Total Process Memory：flink总资源数 2048m，参数：taskmanager.memory.process.size
2. JVM Metaspace：JVM元空间，参数：taskmanager.memory.jvm-metaspace.size，计算：默认值256m
3. JVM Overhead：JVM额外开销，参数：taskmanager.memory.jvm-overhead.fraction，计算：2048 x 0.2=409.6m
4. Total Flink Memory：flink可用资源数，计算：由1-2-3=1.35g
5. Total JVM Heap Memory：JVM堆内存，计算：由4-6=701.44m
6. Total Off-heap Memory：JVM堆外内存，计算：由7+8=680.96m
7. Total JVM Direct Memory：计算：由9+10(默认为0)+11=266.240m
8. Managed：托管内存，参数：taskmanager.memory.flink.size，计算：Total Flink Memory x 0.3=1.35 x 0.3= 414.72m
9. Network：网络缓存，参数：taskmanager.memory.network.fraction，计算：Total Flink Memory x 0.1=1.35 x 0.1= 138.240m
10. Framework：Flink框架内存，参数：taskmanager.memory.framework.heap.size，计算：默认128m
11. Task：任务内存，计算：5-10=573.440m