arm环境下构建Flink的Docker镜像

准备工作

资源准备

按需下载 flink，我的是1.17.2版本。官方说1.13版本之后的安装包兼容了arm架构，所以直接下载就行。

如需要cdc组件，提前下载好。

服务器准备

可在某云上购买arm服务器，2c/4g即可，按量付费。
带宽建议按小时付费（我用的是0.8元/G）。
费用总计：1.35元。
其中流量用了0.99元。
费用没有把控好，下次努力…

镜像准备

准备好可信的jdk-arm镜像。
我这里随便找的一个（houwm/jdk8:arm64），大家谨慎使用。

脚本准备

docker-entrypoint.sh

#!/usr/bin/env bash

###############################################################################
#  Licensed to the Apache Software Foundation (ASF) under one
#  or more contributor license agreements.  See the NOTICE file
#  distributed with this work for additional information
#  regarding copyright ownership.  The ASF licenses this file
#  to you under the Apache License, Version 2.0 (the
#  "License"); you may not use this file except in compliance
#  with the License.  You may obtain a copy of the License at
#
#      http://www.apache.org/licenses/LICENSE-2.0
#
#  Unless required by applicable law or agreed to in writing, software
#  distributed under the License is distributed on an "AS IS" BASIS,
#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
#  See the License for the specific language governing permissions and
# limitations under the License.
###############################################################################

COMMAND_STANDALONE="standalone-job"
COMMAND_HISTORY_SERVER="history-server"

# If unspecified, the hostname of the container is taken as the JobManager address
JOB_MANAGER_RPC_ADDRESS=${JOB_MANAGER_RPC_ADDRESS:-$(hostname -f)}
CONF_FILE="${FLINK_HOME}/conf/flink-conf.yaml"

drop_privs_cmd() {
    if [ $(id -u) != 0 ]; then
        # Don't need to drop privs if EUID != 0
        return
    elif [ -x /sbin/su-exec ]; then
        # Alpine
        echo su-exec flink
    else
        # Others
        echo gosu flink
    fi
}

copy_plugins_if_required() {
  if [ -z "$ENABLE_BUILT_IN_PLUGINS" ]; then
    return 0
  fi

  echo "Enabling required built-in plugins"
  for target_plugin in $(echo "$ENABLE_BUILT_IN_PLUGINS" | tr ';' ' '); do
    echo "Linking ${target_plugin} to plugin directory"
    plugin_name=${target_plugin%.jar}

    mkdir -p "${FLINK_HOME}/plugins/${plugin_name}"
    if [ ! -e "${FLINK_HOME}/opt/${target_plugin}" ]; then
      echo "Plugin ${target_plugin} does not exist. Exiting."
      exit 1
    else
      ln -fs "${FLINK_HOME}/opt/${target_plugin}" "${FLINK_HOME}/plugins/${plugin_name}"
      echo "Successfully enabled ${target_plugin}"
    fi
  done
}

set_config_option() {
  local option=$1
  local value=$2

  # escape periods for usage in regular expressions
  local escaped_option=$(echo ${option} | sed -e "s/\./\\\./g")

  # either override an existing entry, or append a new one
  if grep -E "^${escaped_option}:.*" "${CONF_FILE}" > /dev/null; then
        sed -i -e "s/${escaped_option}:.*/$option: $value/g" "${CONF_FILE}"
  else
        echo "${option}: ${value}" >> "${CONF_FILE}"
  fi
}

prepare_configuration() {
    set_config_option jobmanager.rpc.address ${JOB_MANAGER_RPC_ADDRESS}
    set_config_option blob.server.port 6124
    set_config_option query.server.port 6125

    if [ -n "${TASK_MANAGER_NUMBER_OF_TASK_SLOTS}" ]; then
        set_config_option taskmanager.numberOfTaskSlots ${TASK_MANAGER_NUMBER_OF_TASK_SLOTS}
    fi

    if [ -n "${FLINK_PROPERTIES}" ]; then
        echo "${FLINK_PROPERTIES}" >> "${CONF_FILE}"
    fi
    envsubst < "${CONF_FILE}" > "${CONF_FILE}.tmp" && mv "${CONF_FILE}.tmp" "${CONF_FILE}"
}

maybe_enable_jemalloc() {
    if [ "${DISABLE_JEMALLOC:-false}" == "false" ]; then
        JEMALLOC_PATH="/usr/lib/$(uname -m)-linux-gnu/libjemalloc.so"
        JEMALLOC_FALLBACK="/usr/lib/x86_64-linux-gnu/libjemalloc.so"
        if [ -f "$JEMALLOC_PATH" ]; then
            export LD_PRELOAD=$LD_PRELOAD:$JEMALLOC_PATH
        elif [ -f "$JEMALLOC_FALLBACK" ]; then
            export LD_PRELOAD=$LD_PRELOAD:$JEMALLOC_FALLBACK
        else
            if [ "$JEMALLOC_PATH" = "$JEMALLOC_FALLBACK" ]; then
                MSG_PATH=$JEMALLOC_PATH
            else
                MSG_PATH="$JEMALLOC_PATH and $JEMALLOC_FALLBACK"
            fi
            echo "WARNING: attempted to load jemalloc from $MSG_PATH but the library couldn't be found. glibc will be used instead."
        fi
    fi
}

maybe_enable_jemalloc

copy_plugins_if_required

prepare_configuration

args=("$@")
if [ "$1" = "help" ]; then
    printf "Usage: $(basename "$0") (jobmanager|${COMMAND_STANDALONE}|taskmanager|${COMMAND_HISTORY_SERVER})\n"
    printf "    Or $(basename "$0") help\n\n"
    printf "By default, Flink image adopts jemalloc as default memory allocator. This behavior can be disabled by setting the 'DISABLE_JEMALLOC' environment variable to 'true'.\n"
    exit 0
elif [ "$1" = "jobmanager" ]; then
    args=("${args[@]:1}")

    echo "Starting Job Manager"

    exec $(drop_privs_cmd) "$FLINK_HOME/bin/jobmanager.sh" start-foreground "${args[@]}"
elif [ "$1" = ${COMMAND_STANDALONE} ]; then
    args=("${args[@]:1}")

    echo "Starting Job Manager"

    exec $(drop_privs_cmd) "$FLINK_HOME/bin/standalone-job.sh" start-foreground "${args[@]}"
elif [ "$1" = ${COMMAND_HISTORY_SERVER} ]; then
    args=("${args[@]:1}")

    echo "Starting History Server"

    exec $(drop_privs_cmd) "$FLINK_HOME/bin/historyserver.sh" start-foreground "${args[@]}"
elif [ "$1" = "taskmanager" ]; then
    args=("${args[@]:1}")

    echo "Starting Task Manager"

    exec $(drop_privs_cmd) "$FLINK_HOME/bin/taskmanager.sh" start-foreground "${args[@]}"
fi

args=("${args[@]}")

# Running command in pass-through mode
exec $(drop_privs_cmd) "${args[@]}"

说明：
如果你打好的镜像在启动时提示这个文件有问题，可能是flink版本问题。
你可以从对应版本的x86镜像中，把这个文件捞出来，路径在镜像中的"/"目录下。

Dockerfile

FROM houwm/jdk8:arm64

# Install dependencies
RUN set -ex; \
    apt install -y  tar; \
    apt clean all

ENV GOSU_VERSION=1.17
ENV ARCH=arm64

COPY ./soft/gosu-arm64 /usr/local/bin/gosu
COPY ./soft/flink-1.17.2-bin-scala_2.12.tgz /tmp/flink.tgz

RUN set -ex; \
  export GNUPGHOME="$(mktemp -d)"; \
  chmod +x /usr/local/bin/gosu; \
  gosu nobody true

# Prepare environment
ENV FLINK_HOME=/opt/flink
ENV PATH=$FLINK_HOME/bin:$PATH

RUN groupadd --system --gid=9999 flink && \
    useradd --system --home-dir $FLINK_HOME --uid=9999 --gid=flink flink

WORKDIR $FLINK_HOME

# Install Flink
RUN set -ex; \
  \
  tar -xf /tmp/flink.tgz --strip-components=1; \
  rm /tmp/flink.tgz; \
  \
  chown -R flink:flink .; \
  \
  # Replace default REST/RPC endpoint bind address to use the container's network interface \
  sed -i 's/rest.address: localhost/rest.address: 0.0.0.0/g' $FLINK_HOME/conf/flink-conf.yaml; \
  sed -i 's/rest.bind-address: localhost/rest.bind-address: 0.0.0.0/g' $FLINK_HOME/conf/flink-conf.yaml; \
  sed -i 's/jobmanager.bind-host: localhost/jobmanager.bind-host: 0.0.0.0/g' $FLINK_HOME/conf/flink-conf.yaml; \
  sed -i 's/taskmanager.bind-host: localhost/taskmanager.bind-host: 0.0.0.0/g' $FLINK_HOME/conf/flink-conf.yaml; \
  sed -i '/taskmanager.host: localhost/d' $FLINK_HOME/conf/flink-conf.yaml;

# CDC 包
COPY ./soft/mysql-connector-java-5.1.47.jar /opt/flink/lib/mysql-connector-java.jar
COPY ./soft/flink-connector-jdbc-1.15.3.jar /opt/flink/lib/flink-connector-jdbc.jar
COPY ./soft/flink-sql-connector-oceanbase-cdc-2.2.0.jar /opt/flink/lib/flink-sql-connector-oceanbase-cdc.jar

# Configure container
COPY docker-entrypoint.sh /
RUN chmod +x /docker-entrypoint.sh

ENTRYPOINT ["/docker-entrypoint.sh"]
EXPOSE 6123 8081
CMD ["help"]

说明：
按需修改Dockerfile。

安装Docker

资源更新

apt update
apt upgrade
apt install -y apt-transport-https ca-certificates curl software-properties-common

curl -fsSL http://mirrors.aliyun.com/docker-ce/linux/ubuntu/gpg | sudo apt-key add -

添加源

打开文件/etc/apt/sources.list.d/docker.list,输入下面一行：

deb [arch=arm64] https://mirrors.aliyun.com/docker-ce/linux/debian bullseye stable

再更新一下apt：

apt update

安装docker-ce

apt install docker-ce

添加docker源

在 /etc/docker/daemon.json文件中，添加源：

{
    "registry-mirrors": [
        "https://mirror.ccs.tencentyun.com"
    ]
}

重启docker：

systemctl daemon-reload;systemctl restart docker

构建镜像

目录结构

将相关文件放在主机路径下，目录结构如下：

root@VM-16-11-debian:~# ls -l
-rw-r--r-- 1 root root       5284 Jul 24 16:23 docker-entrypoint.sh
-rw-r--r-- 1 root root       1761 Jul 24 16:33 Dockerfile
drwxr-xr-x 2 root root       4096 Jul 24 15:54 soft

soft目录：

root@VM-16-11-debian:~# ls -l soft/
total 466400
-rw-r--r-- 1 root root 455904906 Jul 24 15:59 flink-1.17.2-bin-scala_2.12.tgz
-rw-r--r-- 1 root root    250075 Jul 24 15:54 flink-connector-jdbc-1.15.3.jar
-rw-r--r-- 1 root root  18016828 Jul 24 15:54 flink-sql-connector-oceanbase-cdc-2.2.0.jar
-rw-r--r-- 1 root root   2402948 Jul 24 15:54 gosu-arm64
-rw-r--r-- 1 root root   1007502 Jul 24 15:54 mysql-connector-java-5.1.47.jar

构建

进入到该目录，执行如下命令构建：

docker build -t flink:1.17 .

成功后查看镜像：

启动验证

创建网络

docker network create flink-network

jobmanager

docker run \
--rm \
--name=jobmanager \
--network flink-network \
--publish 8081:8081 \
--env FLINK_PROPERTIES="jobmanager.rpc.address: jobmanager" \
flink:1.17 jobmanager

taskmanager

docker run \
--rm \
--name=taskmanager \
--network flink-network \
--env FLINK_PROPERTIES="jobmanager.rpc.address: jobmanager" \
flink:1.17 taskmanager

验证

浏览器访问8081端口（注意在防火墙里开启这个端口），能看到flink的控制台即可。

验证成功后可导出镜像：

docker save flink:1.17 -o flink-arm64.tar