Flink_Flink TaskManager 内存模型 ,以及相关内存配置参数

 

Flink 中 TaskManager 中模型比较复杂,本文讲解下 Flink 中的内存模型以及相关的配置 ( 基于 Flink 1.10 )

 

 

内存分布简介 

首先我们看一下 Flink 官网给出的内存示意图 :

https://ci.apache.org/projects/flink/flink-docs-release-1.10/ops/memory/mem_setup.html#set-up-task-executor-memory

 

一般来说我们只需要选择以下3种方式的一种对  task-manager 的内存调整就可以了 ,下面列出这3种方式 :

taskmanager.memory.flink.size
taskmanager.memory.process.size
taskmanager.memory.task.heap.size & taskmanager.memory.managed.size

再结合上面的图,我们给出这3种选配参数之间的关系:

The total process memory of Flink JVM processes consists of memory consumed by Flink application (total Flink memory) and by the JVM to run the process. The total Flink memory consumption includes usage of JVM heap, managed memory (managed by Flink) and other direct (or native) memory.

可以看出来 它们 3者 之间是包含关系  : 

taskmanager.memory.process.size 

>  taskmanager.memory.flink.size 

>  taskmanager.memory.task.heap.size  +   taskmanager.memory.managed.size

 

 

内存分布详细介绍 :

 

但是我们有时候需要更详细的配置调整各个空间的大小 :

下给出 task-manager  更详细的内存分布图

https://ci.apache.org/projects/flink/flink-docs-release-1.10/ops/memory/mem_detail.html#overview

 

The following table lists all memory components, depicted above, and references Flink configuration options which affect the size of the respective components:

下面列出了,所有影响上面内存组件的配置

  Component     Configuration options     Description  
Framework Heap Memory taskmanager.memory.framework.heap.size JVM heap memory dedicated to Flink framework (advanced option)
Task Heap Memory taskmanager.memory.task.heap.size JVM heap memory dedicated to Flink application to run operators and user code
Managed memory taskmanager.memory.managed.size
taskmanager.memory.managed.fraction
Native memory managed by Flink, reserved for sorting, hash tables, caching of intermediate results and RocksDB state backend
Framework Off-heap Memory taskmanager.memory.framework.off-heap.size Off-heap direct (or native) memory dedicated to Flink framework (advanced option)
Task Off-heap Memory taskmanager.memory.task.off-heap.size Off-heap direct (or native) memory dedicated to Flink application to run operators
Network Memory taskmanager.memory.network.min
taskmanager.memory.network.max
taskmanager.memory.network.fraction
Direct memory reserved for data record exchange between tasks (e.g. buffering for the transfer over the network), it is a capped fractionated component of the total Flink memory
JVM metaspace taskmanager.memory.jvm-metaspace.size Metaspace size of the Flink JVM process
JVM Overhead taskmanager.memory.jvm-overhead.min
taskmanager.memory.jvm-overhead.max
taskmanager.memory.jvm-overhead.fraction
Native memory reserved for other JVM overhead: e.g. thread stacks, code cache, garbage collection space etc, it is a capped fractionated component of the total process memory

 

As you can see, the size of some memory components can be simply set by the respective option. Other components can be tuned using multiple options.

如您所见,某些内存组件的大小可以简单地由相应的选项设置。其他组件可以使用多个选项进行自动调整。

 

 

 

 

内存 相关参数的默认值 :

 

Memory Configuration : 

These configuration values control the way that TaskManagers and JobManagers use memory.

Flink tries to shield users as much as possible from the complexity of configuring the JVM for data-intensive processing. In most cases, users should only need to set the values taskmanager.memory.process.size or taskmanager.memory.flink.size (depending on how the setup), and possibly adjusting the ratio of JVM heap and Managed Memory via taskmanager.memory.managed.fraction. The other options below can be used for performane tuning and fixing memory related errors.

For a detailed explanation of how these options interact, see the documentation on TaskManager memory configuration.

Key Default Type Description

taskmanager.memory.flink.size

(none) MemorySize Total Flink Memory size for the TaskExecutors. This includes all the memory that a TaskExecutor consumes, except for JVM Metaspace and JVM Overhead. It consists of Framework Heap Memory, Task Heap Memory, Task Off-Heap Memory, Managed Memory, and Network Memory. See also 'taskmanager.memory.process.size' for total process memory size configuration.

taskmanager.memory.framework.heap.size

128 mb MemorySize Framework Heap Memory size for TaskExecutors. This is the size of JVM heap memory reserved for TaskExecutor framework, which will not be allocated to task slots.

taskmanager.memory.framework.off-heap.size

128 mb MemorySize Framework Off-Heap Memory size for TaskExecutors. This is the size of off-heap memory (JVM direct memory and native memory) reserved for TaskExecutor framework, which will not be allocated to task slots. The configured value will be fully counted when Flink calculates the JVM max direct memory size parameter.

taskmanager.memory.jvm-metaspace.size

256 mb MemorySize JVM Metaspace Size for the TaskExecutors.

taskmanager.memory.jvm-overhead.fraction

0.1 Float Fraction of Total Process Memory to be reserved for JVM Overhead. This is off-heap memory reserved for JVM overhead, such as thread stack space, compile cache, etc. This includes native memory but not direct memory, and will not be counted when Flink calculates JVM max direct memory size parameter. The size of JVM Overhead is derived to make up the configured fraction of the Total Process Memory. If the derived size is less/greater than the configured min/max size, the min/max size will be used. The exact size of JVM Overhead can be explicitly specified by setting the min/max size to the same value.

taskmanager.memory.jvm-overhead.max

1 gb MemorySize Max JVM Overhead size for the TaskExecutors. This is off-heap memory reserved for JVM overhead, such as thread stack space, compile cache, etc. This includes native memory but not direct memory, and will not be counted when Flink calculates JVM max direct memory size parameter. The size of JVM Overhead is derived to make up the configured fraction of the Total Process Memory. If the derived size is less/greater than the configured min/max size, the min/max size will be used. The exact size of JVM Overhead can be explicitly specified by setting the min/max size to the same value.

taskmanager.memory.jvm-overhead.min

192 mb MemorySize Min JVM Overhead size for the TaskExecutors. This is off-heap memory reserved for JVM overhead, such as thread stack space, compile cache, etc. This includes native memory but not direct memory, and will not be counted when Flink calculates JVM max direct memory size parameter. The size of JVM Overhead is derived to make up the configured fraction of the Total Process Memory. If the derived size is less/greater than the configured min/max size, the min/max size will be used. The exact size of JVM Overhead can be explicitly specified by setting the min/max size to the same value.

taskmanager.memory.managed.fraction

0.4 Float Fraction of Total Flink Memory to be used as Managed Memory, if Managed Memory size is not explicitly specified.

taskmanager.memory.managed.size

(none) MemorySize Managed Memory size for TaskExecutors. This is the size of off-heap memory managed by the memory manager, reserved for sorting, hash tables, caching of intermediate results and RocksDB state backend. Memory consumers can either allocate memory from the memory manager in the form of MemorySegments, or reserve bytes from the memory manager and keep their memory usage within that boundary. If unspecified, it will be derived to make up the configured fraction of the Total Flink Memory.

taskmanager.memory.network.fraction

0.1 Float Fraction of Total Flink Memory to be used as Network Memory. Network Memory is off-heap memory reserved for ShuffleEnvironment (e.g., network buffers). Network Memory size is derived to make up the configured fraction of the Total Flink Memory. If the derived size is less/greater than the configured min/max size, the min/max size will be used. The exact size of Network Memory can be explicitly specified by setting the min/max size to the same value.

taskmanager.memory.network.max

1 gb MemorySize Max Network Memory size for TaskExecutors. Network Memory is off-heap memory reserved for ShuffleEnvironment (e.g., network buffers). Network Memory size is derived to make up the configured fraction of the Total Flink Memory. If the derived size is less/greater than the configured min/max size, the min/max size will be used. The exact size of Network Memory can be explicitly specified by setting the min/max to the same value.

taskmanager.memory.network.min

64 mb MemorySize Min Network Memory size for TaskExecutors. Network Memory is off-heap memory reserved for ShuffleEnvironment (e.g., network buffers). Network Memory size is derived to make up the configured fraction of the Total Flink Memory. If the derived size is less/greater than the configured min/max size, the min/max size will be used. The exact size of Network Memory can be explicitly specified by setting the min/max to the same value.

taskmanager.memory.process.size

(none) MemorySize Total Process Memory size for the TaskExecutors. This includes all the memory that a TaskExecutor consumes, consisting of Total Flink Memory, JVM Metaspace, and JVM Overhead. On containerized setups, this should be set to the container memory. See also 'taskmanager.memory.flink.size' for total Flink memory size configuration.

taskmanager.memory.task.heap.size

(none) MemorySize Task Heap Memory size for TaskExecutors. This is the size of JVM heap memory reserved for tasks. If not specified, it will be derived as Total Flink Memory minus Framework Heap Memory, Task Off-Heap Memory, Managed Memory and Network Memory.

taskmanager.memory.task.off-heap.size

0 bytes MemorySize Task Off-Heap Memory size for TaskExecutors. This is the size of off heap memory (JVM direct memory and native memory) reserved for tasks. The configured value will be fully counted when Flink calculates the JVM max direct memory size parameter.

 

 

相关的 JVM 参数 

Flink explicitly adds the following memory related JVM arguments while starting the task executor process, based on the configured or derived memory component sizes:

  JVM Arguments     Value  
-Xmx and -Xms Framework + Task Heap Memory
-XX:MaxDirectMemorySize Framework + Task Off-Heap + Network Memory
-XX:MaxMetaspaceSize JVM Metaspace

 

 

 

常见错误 以及简单示例  

 

由于我使用的是虚拟机器,所以内存有限,我不能使用以上的默认值,我尝试去调整了这些值

 

刚开始由于对内存模型不熟,会遇到如下的问题 :

 exceed configured Total Flink Memory (256.000mb (268435456 bytes)). 

[root@cdh-manager bin]# ./start-cluster.sh 
Starting cluster.
[INFO] 1 instance(s) of standalonesession are already running on cdh-manager.
Starting standalonesession daemon on host cdh-manager.
[ERROR] Unexpected result: 	at org.apache.flink.runtime.util.BashJavaUtils.main(BashJavaUtils.java:46)
[ERROR] The last line of the BashJavaUtils outputs is expected to be the execution result, following the prefix 'BASH_JAVA_UTILS_EXEC_RESULT:'
 - Loading configuration property: jobmanager.rpc.address, localhost
 - Loading configuration property: jobmanager.rpc.port, 6123
 - Loading configuration property: jobmanager.heap.size, 256m
 - Loading configuration property: taskmanager.memory.flink.size, 256m
 - Loading configuration property: taskmanager.numberOfTaskSlots, 1
 - Loading configuration property: parallelism.default, 1
 - Loading configuration property: jobmanager.execution.failover-strategy, region
 - Loading configuration property: taskmanager.memory.network.fraction, 0.1
 - Loading configuration property: taskmanager.memory.network.min, 1mb
 - Loading configuration property: taskmanager.memory.network.max, 256mb
Exception in thread "main" org.apache.flink.configuration.IllegalConfigurationException: Sum of configured Framework Heap Memory (128.000mb (134217728 bytes)), Framework Off-Heap Memory (128.000mb (134217728 bytes)), Task Off-Heap Memory (0 bytes), Managed Memory (102.400mb (107374184 bytes)) and Network Memory (25.600mb (26843546 bytes)) exceed configured Total Flink Memory (256.000mb (268435456 bytes)).
	at org.apache.flink.runtime.clusterframework.TaskExecutorProcessUtils.deriveInternalMemoryFromTotalFlinkMemory(TaskExecutorProcessUtils.java:320)
	at org.apache.flink.runtime.clusterframework.TaskExecutorProcessUtils.deriveProcessSpecWithTotalFlinkMemory(TaskExecutorProcessUtils.java:221)
	at org.apache.flink.runtime.clusterframework.TaskExecutorProcessUtils.processSpecFromConfig(TaskExecutorProcessUtils.java:143)
	at org.apache.flink.runtime.util.BashJavaUtils.getTmResourceJvmParams(BashJavaUtils.java:62)
	at org.apache.flink.runtime.util.BashJavaUtils.main(BashJavaUtils.java:46)
[ERROR] Could not get JVM parameters properly.

 

 

我最终基于我的机器给出了如下的配置 :

taskmanager 的 process memory 为 512m 

内存相关的配置属性 

taskmanager.memory.process.size: 512m

taskmanager.memory.framework.heap.size: 64m
taskmanager.memory.framework.off-heap.size: 64m
taskmanager.memory.jvm-metaspace.size: 64m
taskmanager.memory.jvm-overhead.fraction: 0.2
taskmanager.memory.jvm-overhead.min: 16m
taskmanager.memory.jvm-overhead.max: 64m

taskmanager.memory.network.fraction: 0.1
taskmanager.memory.network.min: 1mb
taskmanager.memory.network.max: 256mb

 

 

 

 

完整的配置文件 flink-conf.yaml

################################################################################
#  Licensed to the Apache Software Foundation (ASF) under one
#  or more contributor license agreements.  See the NOTICE file
#  distributed with this work for additional information
#  regarding copyright ownership.  The ASF licenses this file
#  to you under the Apache License, Version 2.0 (the
#  "License"); you may not use this file except in compliance
#  with the License.  You may obtain a copy of the License at
#
#      http://www.apache.org/licenses/LICENSE-2.0
#
#  Unless required by applicable law or agreed to in writing, software
#  distributed under the License is distributed on an "AS IS" BASIS,
#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
#  See the License for the specific language governing permissions and
# limitations under the License.
################################################################################


#=================================================================
#=================================================================

env.yarn.conf.dir: /etc/hadoop/conf.cloudera.yarn
env.hadoop.conf.dir: /etc/hadoop/conf.cloudera.hdfs



#==============================================================================
# Common
#==============================================================================

# The external address of the host on which the JobManager runs and can be
# reached by the TaskManagers and any clients which want to connect. This setting
# is only used in Standalone mode and may be overwritten on the JobManager side
# by specifying the --host <hostname> parameter of the bin/jobmanager.sh executabl
# In high availability mode, if you use the bin/start-cluster.sh script and setup
# the conf/masters file, this will be taken care of automatically. Yarn/Mesos
# automatically configure the host name based on the hostname of the node where th
# JobManager runs.

jobmanager.rpc.address: cdh-manager

# The RPC port where the JobManager is reachable.

jobmanager.rpc.port: 6123


# The heap size for the JobManager JVM

jobmanager.heap.size: 64m


# The total process memory size for the TaskManager.
#
# Note this accounts for all memory usage within the TaskManager process, includintaspace and other overhead.


taskmanager.memory.process.size: 512m



taskmanager.memory.framework.heap.size: 64m
taskmanager.memory.framework.off-heap.size: 64m
taskmanager.memory.jvm-metaspace.size: 64m
taskmanager.memory.jvm-overhead.fraction: 0.2
taskmanager.memory.jvm-overhead.min: 16m
taskmanager.memory.jvm-overhead.max: 64m



# To exclude JVM metaspace and overhead, please, use total Flink memory size insteaskmanager.memory.process.size'.
# It is not recommended to set both 'taskmanager.memory.process.size' and Flink me
#
# taskmanager.memory.flink.size: 1280m

# The number of task slots that each TaskManager offers. Each slot runs one paralline.

taskmanager.numberOfTaskSlots: 1

# The parallelism used for programs that did not specify and other parallelism.

parallelism.default: 1

# The default file system scheme and authority.
# 
# By default file paths without scheme are interpreted relative to the local
# root file system 'file:///'. Use this to override the default and interpret
# relative paths relative to a different file system,
# for example 'hdfs://mynamenode:12345'
#
# fs.default-scheme

#==============================================================================
# High Availability
#==============================================================================

# The high-availability mode. Possible options are 'NONE' or 'zookeeper'.
#
# high-availability: zookeeper

# The path where metadata for master recovery is persisted. While ZooKeeper stores
# the small ground truth for checkpoint and leader election, this location stores
# the larger objects, like persisted dataflow graphs.
# 
# Must be a durable file system that is accessible from all nodes
# (like HDFS, S3, Ceph, nfs, ...) 
#
# high-availability.storageDir: hdfs:///flink/ha/

# The list of ZooKeeper quorum peers that coordinate the high-availability
# setup. This must be a list of the form:
# "host1:clientPort,host2:clientPort,..." (default clientPort: 2181)
#
# high-availability.zookeeper.quorum: localhost:2181


# ACL options are based on https://zookeeper.apache.org/doc/r3.1.2/zookeeperPrograml#sc_BuiltinACLSchemes
# It can be either "creator" (ZOO_CREATE_ALL_ACL) or "open" (ZOO_OPEN_ACL_UNSAFE)
# The default value is "open" and it can be changed to "creator" if ZK security is
#
# high-availability.zookeeper.client.acl: open

#==============================================================================
# Fault tolerance and checkpointing
#==============================================================================

# The backend that will be used to store operator state checkpoints if
# checkpointing is enabled.
#
# Supported backends are 'jobmanager', 'filesystem', 'rocksdb', or the
# <class-name-of-factory>.
#
# state.backend: filesystem

# Directory for checkpoints filesystem, when using any of the default bundled
# state backends.
#
# state.checkpoints.dir: hdfs://namenode-host:port/flink-checkpoints

# Default target directory for savepoints, optional.
#
# state.savepoints.dir: hdfs://namenode-host:port/flink-checkpoints

# Flag to enable/disable incremental checkpoints for backends that
# support incremental checkpoints (like the RocksDB state backend). 
#
# state.backend.incremental: false

# The failover strategy, i.e., how the job computation recovers from task failures
# Only restart tasks that may have been affected by the task failure, which typicaudes
# downstream tasks and potentially upstream tasks if their produced data is no lonlable for consumption.

jobmanager.execution.failover-strategy: region

#==============================================================================
# Rest & web frontend
#==============================================================================

# The port to which the REST client connects to. If rest.bind-port has
# not been specified, then the server will bind to this port as well.
#
#rest.port: 8081

# The address to which the REST client will connect to
#
#rest.address: 0.0.0.0

# Port range for the REST and web server to bind to.
#
#rest.bind-port: 8080-8090

# The address that the REST & web server binds to
#
#rest.bind-address: 0.0.0.0

# Flag to specify whether job submission is enabled from the web-based
# runtime monitor. Uncomment to disable.

#web.submit.enable: false

#==============================================================================
# Advanced
#==============================================================================

# Override the directories for temporary files. If not specified, the
# system-specific Java temporary directory (java.io.tmpdir property) is taken.
#
# For framework setups on Yarn or Mesos, Flink will automatically pick up the
# containers' temp directories without any need for configuration.
#
# Add a delimited list for multiple directories, using the system directory
# delimiter (colon ':' on unix) or a comma, e.g.:
#     /data1/tmp:/data2/tmp:/data3/tmp
#
# Note: Each directory entry is read from and written to by a different I/O
# thread. You can include the same directory multiple times in order to create
# multiple I/O threads against that directory. This is for example relevant for
# high-throughput RAIDs.
#
io.tmp.dirs: /tmp

# The classloading resolve order. Possible values are 'child-first' (Flink's defau
# and 'parent-first' (Java's default).
#
# Child first classloading allows users to use different dependency/library
# versions in their application than those in the classpath. Switching back
# to 'parent-first' may help with debugging dependency issues.
#
# classloader.resolve-order: child-first

# The amount of memory going to the network stack. These numbers usually need 
# no tuning. Adjusting them may be necessary in case of an "Insufficient number
# of network buffers" error. The default min is 64MB, the default max is 1GB.
# 
# taskmanager.memory.network.fraction: 0.1
# taskmanager.memory.network.min: 64mb
# taskmanager.memory.network.max: 1gb

taskmanager.memory.network.fraction: 0.1
taskmanager.memory.network.min: 1mb
taskmanager.memory.network.max: 256mb

#==============================================================================
# Flink Cluster Security Configuration
#==============================================================================

# Kerberos authentication for various components - Hadoop, ZooKeeper, and connecto
# may be enabled in four steps:
# 1. configure the local krb5.conf file
# 2. provide Kerberos credentials (either a keytab or a ticket cache w/ kinit)
# 3. make the credentials available to various JAAS login contexts
# 4. configure the connector to use JAAS/SASL

# The below configure how Kerberos credentials are provided. A keytab will be used of
# a ticket cache if the keytab path and principal are set.

# security.kerberos.login.use-ticket-cache: true
# security.kerberos.login.keytab: /path/to/kerberos/keytab
# security.kerberos.login.principal: flink-user

# The configuration below defines which JAAS login contexts

# security.kerberos.login.contexts: Client,KafkaClient

#==============================================================================
# ZK Security Configuration
#==============================================================================

# Below configurations are applicable if ZK ensemble is configured for security

# Override below configuration to provide custom ZK service name if configured
# zookeeper.sasl.service-name: zookeeper

# The configuration below must match one of the values set in "security.kerberos.ltexts"
# zookeeper.sasl.login-context-name: Client

#==============================================================================
# HistoryServer
#==============================================================================

# The HistoryServer is started and stopped via bin/historyserver.sh (start|stop)

# Directory to upload completed jobs to. Add this directory to the list of
# monitored directories of the HistoryServer as well (see below).
#jobmanager.archive.fs.dir: hdfs:///completed-jobs/

# The address under which the web-based HistoryServer listens.
#historyserver.web.address: 0.0.0.0

# The port under which the web-based HistoryServer listens.
#historyserver.web.port: 8082

# Comma separated list of directories to monitor for completed jobs.
#historyserver.archive.fs.dir: hdfs:///completed-jobs/

# Interval in milliseconds for refreshing the monitored directories.
#historyserver.archive.fs.refresh-interval: 10000

 

 

 

 

展开阅读全文

没有更多推荐了,返回首页

©️2019 CSDN 皮肤主题: 编程工作室 设计师: CSDN官方博客
应支付0元
点击重新获取
扫码支付

支付成功即可阅读