flink中的内存划分-CSDN博客

在设置完taskManager内存之后相当于向yarn申请这么大内存的container，然后flink内部的内存大部分是由flink框架管理，在启动container之前就会预先计算各个内存块的大小。

内存块划分


// 默认值0.25f
final float memoryCutoffRatio = config.getFloat(ResourceManagerOptions.CONTAINERIZED_HEAP_CUTOFF_RATIO);

// 最少预留大小默认600MB
final int minCutoff = config.getInteger(ResourceManagerOptions.CONTAINERIZED_HEAP_CUTOFF_MIN);
            
// 先减去一块内存预留给jvm
long cutoff = (long) (containerMemoryMB * memoryCutoffRatio);
if (cutoff < minCutoff) {
    cutoff = minCutoff;
}

final long javaMemorySizeMB = containerMemoryMB - cutoff;

// (2) split the remaining Java memory between heap and off-heap
final long heapSizeMB = TaskManagerServices.calculateHeapSizeMB(javaMemorySizeMB, config);
// use the cut-off memory for off-heap (that was its intention)
// 计算得到堆外内存后，总内存减去得到堆内的大小
final long offHeapSize = javaMemorySizeMB == heapSizeMB ? -1L : containerMemoryMB - heapSizeMB;

// (3) obtain the additional environment variables from the configuration
final HashMap<String, String> envVars = new HashMap<>();
final String prefix = ResourceManagerOptions.CONTAINERIZED_TASK_MANAGER_ENV_PREFIX;

for (String key : config.keySet()) {
    if (key.startsWith(prefix) && key.length() > prefix.length()) {
        // remove prefix
        String envVarKey = key.substring(prefix.length());
        envVars.put(envVarKey, config.getString(key, null));
    }
}	
复制代码

计算堆内内存大小`TaskManagerServices.calculateHeapSizeMB`


// 默认线上是开启堆外内存的，为了数据交换的过程只使用堆外内存，gc友好
if (useOffHeap) {
    // subtract the Java memory used for network buffers
    final long networkBufMB = calculateNetworkBufferMemory(totalJavaMemorySize, config) >> 20; // bytes to megabytes
    final long remainingJavaMemorySizeMB = totalJavaMemorySizeMB - networkBufMB;

    long offHeapSize = config.getLong(TaskManagerOptions.MANAGED_MEMORY_SIZE);

    if (offHeapSize <= 0) {
        // calculate off-heap section via fraction
        // 将划去networkBuffer大小*一个堆外的系数（默认是0.7）得到其他的堆外内存
        double fraction = config.getFloat(TaskManagerOptions.MANAGED_MEMORY_FRACTION);
        offHeapSize = (long) (fraction * remainingJavaMemorySizeMB);
    }

    TaskManagerServicesConfiguration
        .checkConfigParameter(offHeapSize < remainingJavaMemorySizeMB, offHeapSize,
            TaskManagerOptions.MANAGED_MEMORY_SIZE.key(),
            "Managed memory size too large for " + networkBufMB +
                " MB network buffer memory and a total of " + totalJavaMemorySizeMB +
                " MB JVM memory");

    heapSizeMB = remainingJavaMemorySizeMB - offHeapSize;
} else {
    heapSizeMB = totalJavaMemorySizeMB;
}
复制代码

计算堆外内存大小`calculateNetworkBufferMemory`

Preconditions.checkArgument(totalJavaMemorySize > 0);

int segmentSize = config.getInteger(TaskManagerOptions.MEMORY_SEGMENT_SIZE);

final long networkBufBytes;
// 涉及新老两个版本的参数，以前版本是直接设置networkbuffer的个数，但是比较难估计，新的版本是直接设置内存块大小
if (TaskManagerServicesConfiguration.hasNewNetworkBufConf(config)) {
    // new configuration based on fractions of available memory with selectable min and max
    float networkBufFraction = config.getFloat(TaskManagerOptions.NETWORK_BUFFERS_MEMORY_FRACTION);
    long networkBufMin = config.getLong(TaskManagerOptions.NETWORK_BUFFERS_MEMORY_MIN);
    long networkBufMax = config.getLong(TaskManagerOptions.NETWORK_BUFFERS_MEMORY_MAX);

    TaskManagerServicesConfiguration
        .checkNetworkBufferConfig(segmentSize, networkBufFraction, networkBufMin, networkBufMax);

    // 通过networkFraction计算network的内存大小，这个fraction默认值是0.1，同时最小设置默认是64M
    networkBufBytes = Math.min(networkBufMax, Math.max(networkBufMin,
        (long) (networkBufFraction * totalJavaMemorySize)));

    TaskManagerServicesConfiguration
        .checkConfigParameter(networkBufBytes < totalJavaMemorySize,
            "(" + networkBufFraction + ", " + networkBufMin + ", " + networkBufMax + ")",
            "(" + TaskManagerOptions.NETWORK_BUFFERS_MEMORY_FRACTION.key() + ", " +
                TaskManagerOptions.NETWORK_BUFFERS_MEMORY_MIN.key() + ", " +
                TaskManagerOptions.NETWORK_BUFFERS_MEMORY_MAX.key() + ")",
            "Network buffer memory size too large: " + networkBufBytes + " >= " +
                totalJavaMemorySize + " (total JVM memory size)");
} else {
    // use old (deprecated) network buffers parameter
    int numNetworkBuffers = config.getInteger(TaskManagerOptions.NETWORK_NUM_BUFFERS);
    networkBufBytes = (long) numNetworkBuffers * (long) segmentSize;

    TaskManagerServicesConfiguration.checkNetworkConfigOld(numNetworkBuffers);

    TaskManagerServicesConfiguration
        .checkConfigParameter(networkBufBytes < totalJavaMemorySize,
            networkBufBytes, TaskManagerOptions.NETWORK_NUM_BUFFERS.key(),
            "Network buffer memory size too large: " + networkBufBytes + " >= " +
                totalJavaMemorySize + " (total JVM memory size)");
}

return networkBufBytes;
复制代码

小结

JVM预留内存，总内存的20%，最小预留，600M
剩下的内存的10%作为networkBuffer的内存，最小64M
剩下内存30%设为堆内内存，总内存减去堆内内存设为directMemory，用于netty和rocksDB和networkBuffer以及JVM自身内存

举例

一个启动时设置TaskManager内存大小为1024MB

1024MB - (1024 * 0.2 < 600MB) -> 600MB = 424MB (cutoff)
424MB - (424MB * 0.1 < 64MB) -> 64MB = 360MB (networkbuffer)
360MB * (1 - 0.7) = 108MB -> （onHeap）
1024MB - 108MB = 916MB （maxDirectMemory）

最终启动命令：

yarn      46218  46212  1 Jan08 ?        00:17:50
/home/yarn/java-current/bin/java 
-Xms109m -Xmx109m -XX:MaxDirectMemorySize=915m -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps 
-XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=2 -XX:GCLogFileSize=512M 
-Xloggc:/data1/hadoopdata/nodemanager/logdir/application_1545981373722_0172/container_e194_1545981373722_0172_01_000005/taskmanager_gc.log 
-XX:+UseConcMarkSweepGC 
-XX:CMSInitiatingOccupancyFraction=75 
-XX:+UseCMSInitiatingOccupancyOnly 
-XX:+AlwaysPreTouch -server 
-XX:+HeapDumpOnOutOfMemoryError 
-Dlog.file=/data1/hadoopdata/nodemanager/logdir/application_1545981373722_0172/container_e194_1545981373722_0172_01_000005/taskmanager.log 
-Dlogback.configurationFile=file:./logback.xml 
-Dlog4j.configuration=file:./log4j.properties org.apache.flink.yarn.YarnTaskManager 
--configDir .
复制代码