arm64 环境利用 docker 编译redis filebeat airflow hue 以及 alertmanager 镜像

先准备一个编译环境压压惊

• 首先要有一个 docker

部署 docker

wget https://download.docker.com/linux/static/stable/aarch64/docker-19.03.11.tgz
tar xvf docker-19.03.11.tgz
mv docker/* /usr/bin/

docker.service

cat <<EOF> /usr/lib/systemd/system/docker.service
[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
BindsTo=containerd.service
After=network-online.target firewalld.service containerd.service
Wants=network-online.target
Requires=docker.socket
 
[Service]
Type=notify
# the default is not to use systemd for cgroups because the delegate issues still
# exists and systemd currently does not support the cgroup feature set required
# for containers run by docker
ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
ExecReload=/bin/kill -s HUP \$MAINPID
TimeoutSec=0
RestartSec=2
Restart=always
 
# Note that StartLimit* options were moved from "Service" to "Unit" in systemd 229.
# Both the old, and new location are accepted by systemd 229 and up, so using the old location
# to make them work for either version of systemd.
StartLimitBurst=3
 
# Note that StartLimitInterval was renamed to StartLimitIntervalSec in systemd 230.
# Both the old, and new name are accepted by systemd 230 and up, so using the old name to make
# this option work for either version of systemd.
StartLimitInterval=60s
 
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
 
# Comment TasksMax if your systemd version does not support it.
# Only systemd 226 and above support this option.
TasksMax=infinity
# set delegate yes so that systemd does not reset the cgroups of docker containers
Delegate=yes
 
# kill only the docker process, not all processes in the cgroup
KillMode=process
 
[Install]
WantedBy=multi-user.target
EOF

docker.socket

cat <<EOF> /usr/lib/systemd/system/docker.socket
[Unit]
Description=Docker Socket for the API
PartOf=docker.service
 
[Socket]
ListenStream=/var/run/docker.sock
SocketMode=0660
SocketUser=root
SocketGroup=docker
 
[Install]
WantedBy=sockets.targe
EOF

containerd.service

cat <<EOF> /usr/lib/systemd/system/containerd.service
#   Copyright 2018-2020 Docker Inc.
 
#   Licensed under the Apache License, Version 2.0 (the "License");
#   you may not use this file except in compliance with the License.
#   You may obtain a copy of the License at
 
#       http://www.apache.org/licenses/LICENSE-2.0
 
#   Unless required by applicable law or agreed to in writing, software
#   distributed under the License is distributed on an "AS IS" BASIS,
#   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
#   See the License for the specific language governing permissions and
#   limitations under the License.
 
[Unit]
Description=containerd container runtime
Documentation=https://containerd.io
After=network.target
 
[Service]
ExecStartPre=-/sbin/modprobe overlay
ExecStart=/usr/bin/containerd
KillMode=process
Delegate=yes
LimitNOFILE=1048576
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNPROC=infinity
LimitCORE=infinity
TasksMax=infinity
 
[Install]
WantedBy=multi-user.target
EOF

daemon.json

groupadd docker
mkdir /etc/docker/
cat > /etc/docker/daemon.json <<EOF
   {
    "live-restore": true,
    "exec-opts": ["native.cgroupdriver=systemd"],
    "log-driver": "json-file",
    "registry-mirrors": ["https://v16stybc.mirror.aliyuncs.com"],
    "log-opts": {
       "max-size": "100m"
    },
    "storage-driver": "overlay2",
    "storage-opts": [
    "overlay2.override_kernel_check=true"
    ]
    }
EOF

准备镜像

FROM centos:7
ARG work_dir=/usr/local

ENV PS1='\[\e[7;34m\]\u@\h\[\e[0m\]\[\e[0;35m\]:$(pwd) \[\e[0m\]\[\e[0;35m\]\t\[\e[0m\]\n\[\e[0;32m\]> \[\e[0m\]'
ENV LANG=en_US.UTF-8
ENV TZ="Asia/Shanghai"
ENV GOPATH=/usr/local/src
ENV GOPROXY=https://goproxy.cn
ENV GO_HOME=${work_dir}/go
ENV PATH=${PATH}:${GO_HOME}/bin

WORKDIR ${work_dir}
ADD ./go1.17.9.linux-arm64.tar.gz ./
RUN curl -o /etc/yum.repos.d/ http://mirrors.aliyun.com/repo/Centos-altarch-7.repo  && \
    yum install -y wget vim unzip bash-completion git gcc gcc-c++ make cmake protobuf-* automake libtool zlib-devel openssl-devel && \
    yum clean all && \
    echo '/usr/bin/sleep 315360000' > start.sh && \
    chmod +x start.sh
CMD ["/usr/bin/bash","start.sh"]

以 host 网络模式来 build docker 镜像

nat 模式局限性比较大，很多网络会超时

docker build --network host -t go_make:arm64_1.17.9 .

把容器放后台启动，方便进进出出

docker run -d --network host --name make_some_thing go_make:arm64_1.17.9

进入容器

docker exec -it make_some_thing bash

修改 go 模块的下载地址为国内

go env -w GOPROXY=https://goproxy.cn

编译 redis

• redis 官网下载地址

wget https://download.redis.io/releases/redis-4.0.14.tar.gz
tar xvf redis-4.0.14.tar.gz
cd redis-4.0.14/
make
make install

查看 redis 版本，验证编译是否成功

./src/redis-cli --version

其他细节

`GLIBC_2.28’ not found

• 编译好的 redis-server 拿到其他环境，可能会出现的问题

redis-server: /lib64/libc.so.6: version `GLIBC_2.28' not found

GLIBC 2.8 对于 make 的版本有要求，不能低于 4.x ，我们这边用 4.2 版本的

不升级 make ，编译的时候会报错 These critical programs are missing or too old: make bison compiler

编译 make

wget https://ftp.gnu.org/gnu/make/make-4.2.tar.gz --no-check-certificate
tar xvf make-4.2.tar.gz
cd make-4.2/
./configure && make -j $(nproc) && make install
mv /usr/bin/make{,-$(make --version | head -1 | awk '{print $NF}')}
mv make /usr/bin/

查看 make 版本，验证编译是否成功

make --version

编译 glibc 2.8

• 升级 gcc
  • These critical programs are missing or too old: bison compiler
  • 编译的时候会有上面的报错表示 gcc 版本太低了，最少要 gcc-7

yum install -y centos-release-scl bison
yum install -y devtoolset-7-gcc devtoolset-7-gcc-c++ devtoolset-7-binutils
# 使环境生效，通过 gcc -v 可以查看版本已经变成 7.x了
scl enable devtoolset-7 bash
# 永久生效
echo "source /opt/rh/devtoolset-7/enable" >>/etc/profile
source /etc/profile

• 编译 glibc 2.8

切记，glibc 2.8 不要在 /usr/local 目录下编译，编译的时候会有下面这样的提示，要使用 --prefix 指定路径

*** On GNU/Linux systems the GNU C Library should not be installed into
*** /usr/local since this might make your system totally unusable.
*** We strongly advise to use a different prefix.  For details read the FAQ.
*** If you really mean to do this, run configure again using the extra
*** parameter `--disable-sanity-checks'.

wget https://ftp.gnu.org/gnu/glibc/glibc-2.28.tar.xz --no-check-certificate
tar xvf glibc-2.28.tar.xz
cd glibc-2.28
# 为什么要建一个新目录？
## 因为直接执行'./configure'，会报错'configure: error: you must configure in a separate build directory'
mkdir build
cd build/
mkdir /lib64/glibc-2.28/etc
# 可能会遇到这个报错：Warning: ignoring configuration file that cannot be opened: /lib64/glibc-2.28/etc/ld.so.conf: No such file or directory
## find 查找一下文件，然后做个软连接就可以了
ln -s $(find / -name "ld.so.conf") /lib64/glibc-2.28/etc/ld.so.conf
../configure --prefix=/lib64/glibc-2.28 && make -j $(nproc) && make install

编译 filebeat

• beats github

wget https://github.com/elastic/beats/archive/refs/tags/v7.7.0.tar.gz
tar xvf v7.7.0.tar.gz
cd beats-7.7.0/filebeat
make

执行的时候会有一段时间终端没有任务输出，耐心等待就可以了，输出类似如下内容，表示编译完成

fatal: Not a git repository (or any of the parent directories): .git
go build -ldflags "-X github.com/elastic/beats/libbeat/version.buildTime=2022-04-29T08:21:45Z -X github.com/elastic/beats/libbeat/version.commit="
go: downloading github.com/imdario/mergo v0.3.6
go: downloading github.com/urso/go-bin v0.0.0-20180220135811-781c575c9f0e
go: downloading github.com/google/gofuzz v1.0.0
go: downloading github.com/davecgh/go-spew v1.1.1
go: downloading github.com/google/go-cmp v0.4.0
go: downloading github.com/json-iterator/go v1.1.7
go: downloading github.com/modern-go/reflect2 v1.0.1
go: downloading google.golang.org/grpc v1.27.1
go: downloading github.com/containerd/containerd v1.3.3
go: downloading github.com/sirupsen/logrus v1.4.2
go: downloading github.com/docker/distribution v2.7.1+incompatible
go: downloading github.com/eapache/queue v1.1.0
go: downloading github.com/pierrec/lz4 v2.2.6+incompatible
go: downloading github.com/eapache/go-xerial-snappy v0.0.0-20180814174437-776d5712da21
go: downloading github.com/eapache/go-resiliency v1.1.0
go: downloading github.com/hashicorp/go-uuid v1.0.1
go: downloading gopkg.in/jcmturner/dnsutils.v1 v1.0.1
go: downloading gopkg.in/jcmturner/aescts.v1 v1.0.1
go: downloading gopkg.in/jcmturner/rpc.v1 v1.1.0
go: downloading github.com/golang/snappy v0.0.1
go: downloading github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd
go: downloading google.golang.org/genproto v0.0.0-20191230161307-f3c370f40bfb

查看 filebeat 版本，验证编译是否成功

./filebeat version

从容器内打包的时候，需要注意一个细节，在宿主机上执行 file filebeat 命令会看到有 lib 文件依赖，在其他环境使用的时候，也要注意把 lib 文件复制过去

通过 file 命令可以看到，依赖的 lib 文件的路径和名称为 /lib/ld-linux-aarch64.so.1

filebeat-7.7.0-linux-arm64/filebeat: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-aarch64.so.1, for GNU/Linux 3.7.0, Go BuildID=2ZFlNlPqOIS1j0hZOMLt/zmkml0JZODmFUTnWldZe/wMGZe6JzkJmCcs5EeF-6/ID26ze0hYOJZMk5gMX19, BuildID[sha1]=438f66d9c4384ccde3a4a395034160565df6280e, not stripped

安装 airflow

yum install -y python3 python3-devel python3-pip
pip3 install --upgrade pip
pip3 install apache-airflow==2.2.3

升级 sqlite3

• airflow 2.2.3 环境要求
• sqlite 下载地址

官方要求 airflow 2.0+ 版本的环境， python 的 sqlite3 版本不能低于 3.15.0

arm 环境下载的 python 3.6 使用的 sqlite3 版本是 3.7.17 的，运行 airflow --version 会报错 error: sqlite C library version too old (< 3.15.0)

Python 3.6.8 (default, Nov 16 2020, 16:33:14)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-44)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sqlite3
>>> sqlite3.sqlite_version
'3.7.17'

wget --no-check-certificate https://sqlite.org/2022/sqlite-autoconf-3380400.tar.gz
tar xvf sqlite-autoconf-3380400.tar.gz
cd sqlite-autoconf-3380400/
export CFLAGS="-DSQLITE_ENABLE_FTS3 \
    -DSQLITE_ENABLE_FTS3_PARENTHESIS \
    -DSQLITE_ENABLE_FTS4 \
    -DSQLITE_ENABLE_FTS5 \
    -DSQLITE_ENABLE_JSON1 \
    -DSQLITE_ENABLE_LOAD_EXTENSION \
    -DSQLITE_ENABLE_RTREE \
    -DSQLITE_ENABLE_STAT4 \
    -DSQLITE_ENABLE_UPDATE_DELETE_LIMIT \
    -DSQLITE_SOUNDEX \
    -DSQLITE_TEMP_STORE=3 \
    -DSQLITE_USE_URI \
    -O2 \
    -fPIC"
export PREFIX="/usr/local/sqlite3"
LIBS="-lm" ./configure --disable-tcl --enable-shared --enable-tempstore=always --prefix="$PREFIX"
make -j $(nproc)
make install

这里的 /usr/local 要和上面编译的时候 --prefix 路径一致

export LD_LIBRARY_PATH=/usr/local/sqlite3/lib:$LD_LIBRARY_PATH

查看 airflow 版本，验证编译是否成功

airflow version

编译 hue

安装相关依赖

• 编译 hue 需要安装 python-devel 否则 make 的时候会报错：/usr/local/hue-release-4.7.1/Makefile.vars:65: *** "Error: must have python development packages for python2.7. Could not find Python.h. Please install python2.7-devel". Stop
• 不安装 libffi-devel ，pip 安装 cffi 的时候会报错：fatal error: ffi.h: No such file or directory
• 不安装 MySQL-python，pip 安装 mysql-python 会报错：EnvironmentError: mysql_config not found
• 不安装 mysql-devel，pip 安装 mysql-python 会报错：EnvironmentError: mysql_config not found
• 不安装 sqlite-devel ，编译的时候会报错：fatal error: sqlite3.h: No such file or directory
• 不安装 cyrus-sasl-devel，编译的时候会报错：fatal error: sasl/sasl.h: No such file or directory
• 不安装 openldap-devel，编译的时候会报错：fatal error: lber.h: No such file or directory
• 不安装 libxslt-devel，编译的时候会报错：fatal error: libxml/xmlversion.h: No such file or directory

yum install -y python-devel \
libffi-devel \
MySQL-python \
mysql-devel \
sqlite-devel \
cyrus-sasl-devel \
openldap-devel \
libxslt-devel

安装 nodejs

如何确定 nodejs 的版本

下载好 hue 的安装包，解压后进入 hue 的路径下，执行下面的命令

grep setup tools/container/base/hue/Dockerfile

curl -sL https://rpm.nodesource.com/setup_10.x 这里就可以看到，用到的是 10.x 的版本

wget https://nodejs.org/dist/v10.19.0/node-v10.19.0-linux-arm64.tar.gz
tar xvf node-v10.19.0-linux-arm64.tar.gz
echo 'export NODE_HOME=/usr/local/node-v10.19.0-linux-arm64' >> /etc/profile
echo 'export PATH=$PATH:$NODE_HOME/bin' >> /etc/profile
source /etc/profile
# 验证环境变量是否生效
node -v
npm -v

开始编译

wget https://github.com/cloudera/hue/archive/refs/tags/release-4.7.1.tar.gz
tar xvf release-4.7.1.tar.gz
cd hue-release-4.7.1/
build/env/bin/python2.7 -m pip install cffi \
mysql-python \
traitlets \
backports.shutil-get-terminal-size \
pathlib2 \
pexpect \
pickleshare \
simplegeneric==0.8.1 \
prompt-toolkit==1.0.4 
make apps

通过 cat tools/container/base/hue/Dockerfile 可以看到 hue 的编译需要解决的依赖，但是我为了让镜像更小，能不装的就不装，所以在一步一步的试错

报错1

distutils.errors.DistutilsError: Command '['/usr/local/hue-release-4.7.1/build/env/bin/python2.7', '-m', 'pip', '--disable-pip-version-check', 'wheel', '--no-deps', '-w', '/usr/local/hue-release-4.7.1/desktop/core/ext-py/cryptography-2.9/temp/tmpojt9TZ', '--quiet', 'cffi!=1.11.3,>=1.8']' returned non-zero exit status 1

build/env/bin/python2.7 -m pip install cffi

报错2

sh: mysql_config: command not found

build/env/bin/python2.7 -m pip install mysql-python

报错3

File "/tmp/easy_install-_GDIGO/traitlets-5.1.1/setup.py", line 41 print(error, file=sys.stderr)

build/env/bin/python -m pip install traitlets

报错4

ERROR: pip's legacy dependency resolver does not consider dependency conflicts when selecting packages. This behaviour is the source of the following dependency conflicts. ipython 5.2.0 requires backports.shutil-get-terminal-size, which is not installed. ipython 5.2.0 requires pathlib2, which is not installed. ipython 5.2.0 requires pexpect, which is not installed. ipython 5.2.0 requires pickleshare, which is not installed. ipython 5.2.0 requires prompt-toolkit<2.0.0,>=1.0.4, which is not installed. ipython 5.2.0 requires simplegeneric>0.8, which is not installed.

build/env/bin/python -m pip install backports.shutil-get-terminal-size \
pathlib2 \
pexpect \
pickleshare \
prompt-toolkit==1.0.4 \
simplegeneric==0.8.1 

制作 alertmanager 镜像

获取二进制文件

alertmanager github

wget https://github.com/prometheus/alertmanager/releases/download/v0.14.0/alertmanager-0.14.0.linux-arm64.tar.gz
tar xvf alertmanager-0.14.0.linux-arm64.tar.gz
mv alertmanager-0.14.0.linux-arm64/a* .
vim config.yaml

route:
  group_by: ['alertname']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 1h
  receiver: 'web.hook'
receivers:
  - name: 'web.hook'
    webhook_configs:
      - url: 'http://127.0.0.1:5001/'
inhibit_rules:
  - source_match:
      severity: 'critical'
    target_match:
      severity: 'warning'
    equal: ['alertname', 'dev', 'instance']

利用 busybox 来制作 alertmanager 镜像 （因为 busybox 很小）

FROM busybox:latest

COPY amtool /bin/amtool
COPY alertmanager /bin/alertmanager
COPY config.yml /etc/alertmanager/config.yml

RUN mkdir -p /alertmanager && \
    chown -R nobody:nobody etc/alertmanager /alertmanager

USER       nobody
EXPOSE     9093
VOLUME     [ "/alertmanager" ]
WORKDIR    /alertmanager
ENTRYPOINT [ "/bin/alertmanager" ]
CMD        [ "--config.file=/etc/alertmanager/config.yml", \
             "--storage.path=/alertmanager" ]

docker build -t prom/alertmanager:v0.14.0 .