1、Flink TaskManager启动日志
INFO [] - Loading configuration property: taskmanager.memory.process.size, 2048m
INFO [] - Loading configuration property: taskmanager.memory.managed.fraction, 0.3
INFO [] - Loading configuration property: taskmanager.memory.jvm-overhead.fraction, 0.2
INFO [] - Loading configuration property: taskmanager.memory.network.fraction, 0.1
INFO [] - Loading configuration property: parallelism.default, 2
INFO [] - Loading configuration property: task.cancellation.timeout, 0
INFO [] - Final TaskExecutor Memory configuration:
INFO [] - Total Process Memory: 2.000gb (2147483648 bytes)
INFO [] - Total Flink Memory: 1.350gb (1449551456 bytes)
INFO [] - Total JVM Heap Memory: 701.440mb (735513127 bytes)
INFO [] - Framework: 128.000mb (134217728 bytes)
INFO [] - Task: 573.440mb (601295399 bytes)
INFO [] - Total Off-heap Memory: 680.960mb (714038329 bytes)
INFO [] - Managed: 414.720mb (434865454 bytes)
INFO [] - Total JVM Direct Memory: 266.240mb (279172875 bytes)
INFO [] - Framework: 128.000mb (134217728 bytes)
INFO [] - Task: 0 bytes
INFO [] - Network: 138.240mb (144955147 bytes)
INFO [] - JVM Metaspace: 256.000mb (268435456 bytes)
INFO [] - JVM Overhead: 409.600mb (429496736 bytes)
- Total Process Memory:flink总资源数 2048m,参数:taskmanager.memory.process.size
- JVM Metaspace:JVM元空间,参数:taskmanager.memory.jvm-metaspace.size,计算:默认值256m
- JVM Overhead:JVM额外开销,参数:taskmanager.memory.jvm-overhead.fraction,计算:2048 x 0.2=409.6m (一般这块内存可以调小)
- Total Flink Memory:flink可用资源数,计算:由2048-256-409=1.35g
- Total JVM Heap Memory:JVM堆内存,计算:由1350-680=701.44m
- Total Off-heap Memory:JVM堆外内存,计算:由414+268=680.96m
- Total JVM Direct Memory:计算:由9+10(默认为0)+11=266.240m
- Managed:托管内存,参数:taskmanager.memory.flink.size,计算:Total Flink Memory x 0.3=1.35 x 0.3= 414.72m
- Network:网络缓存,参数:taskmanager.memory.network.fraction,计算:Total Flink Memory x 0.1=1.35 x 0.1= 138.240m
- Framework:Flink框架内存,参数:taskmanager.memory.framework.heap.size,计算:默认128m
- Task:任务内存,计算:5-10=573.440m
下面更好表达
metaspace 与overhead区别 :metaspace可以清晰管理,有参数可以控制大小
NetworkBuffers:
使用Direct Memory
用于数据传输缓冲
特点:
同一Task Excutor内各slot之间没有隔离
需要多少有作业拓扑决定,不足时导致任务失败
MemoryManager:
使用Native Memory
用于RockesDBStateBackend和Batch 0prator (sort 和 join 等操作符)
HeapStateBackend or 无状态 可以将Memory memory内存设置为0,防止浪费资源
参考
https://mp.weixin.qq.com/s/K65OjUNM1Wf8Z5H9GS4zbg
https://mp.weixin.qq.com/s/ZTN9AaUFvhrRBz-WC-6a0A
https://mp.weixin.qq.com/s/ltzPIdJWOR_drnambp0XcQ
https://files.alicdn.com/tpsservice/303974c40369839ed6dc175664237a54.pdf