ARM64上Docker编译问题与解决

最新推荐文章于 2024-07-29 09:38:22 发布

weixin_33862993

最新推荐文章于 2024-07-29 09:38:22 发布

阅读量973

点赞数

文章标签：运维嵌入式 golang

原文链接：https://my.oschina.net/u/2306127/blog/803599

版权

2019独角兽企业重金招聘Python工程师标准>>>

编译过程

在96Boards 的410C上编译Docker（这里使用Dokcer 1.14 Master版本，方法参考），遇到些问题，解决方法如下：

修改Docker源码目录下的Dockerfile.aarch64文件

修改 FROM aarch64/ubuntu:wily 去掉wily，因为wily版很多源都没法下载。
Docker编译需要GOROOT_BOOTSTRAP，在安装包中加入： golang
在docker-py安装前运行：pip install setuptools
需要下载go的源代码安装，访问不了的设置代理，示意：curl -x 192.168.1.11:9999，把这里的地址改成自己的代理服务器。
内存不足，启用虚存支持，参考：https://my.oschina.net/u/2306127/blog/657269
主存储空间不足，把Docker放到U盘或SD卡上，参考：https://my.oschina.net/u/2306127/blog/653569

** 基于ARM64的很多容器资源：https://github.com/multiarch**

运行问题解决

因为不明原因，最后make deb编译失败，但到目录中看，bundles中dockerD和docker都已经生成好了。

拷贝到/usr/bin目录中，修改/lib/systemd/system/docker.service文件。
但systemctl docker start运行时又报container找不到错误。
到 **hack/dockerfile/install-binaries.sh **中找到containerd和runc的相应安装代码，手工运行、安装到/usr/local/bin，建立软连接，docker就可以正常启动了。

注意：

由于最近github.com强制使用了https，导致go get和git clone 原来的http网站失败。
出错显示，网站找不到，参考这里的方法解决。

SSL CA错误的解决：
运行：git config --global http.sslverify false.
或者：env GIT_SSL_NO_VERIFY=true
添加CA：http://askubuntu.com/questions/645818/how-to-install-certificates-for-command-line

安装containerd和runc

export GOPATH="/home/linaro" 

#复制DockerD服务器到系统目录
cp docker1.14.0-dev /usr/bin/
#修改/lib/systemd/system/docker.service中的ExecStart=/usr/bin/dockerd为ExecStart=/usr/bin/docker1.14.0-dev

#以后使用dox来运行新的Docker客户端。
cp  docker1.14.0 /usr/local/bin/dox
 
# Install runc
export RUNC_COMMIT=f59ba3cdd76fdc08c004f42aa915996f6f420899
git clone https://github.com/docker/runc.git "$GOPATH/src/github.com/opencontainers/runc"
cd "$GOPATH/src/github.com/opencontainers/runc"
git checkout -q "$RUNC_COMMIT"
make static BUILDTAGS="seccomp apparmor selinux" 
cp runc /usr/local/bin/docker-runc
ln -s /usr/local/bin/docker-runc /usr/local/bin/runc

# Install containerd
export CONTAINERD_COMMIT=2a5e70cbf65457815ee76b7e5dd2a01292d9eca8
git clone https://github.com/docker/containerd.git "$GOPATH/src/github.com/docker/containerd" 
cd "$GOPATH/src/github.com/docker/containerd"
git checkout -q "$CONTAINERD_COMMIT"
make static
cp bin/containerd /usr/local/bin/docker-containerd
ln -s /usr/local/bin/docker-containerd  /usr/local/bin/containerd

cp bin/containerd-shim /usr/local/bin/docker-containerd-shim
ln -s /usr/local/bin/docker-containerd-shim /usr/local/bin/containerd-shim

cp bin/ctr /usr/local/bin/docker-containerd-ctr
ln -s /usr/local/bin/docker-containerd-ctr /usr/local/bin/containerd-ctr

修改的docker.service文件

[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
After=network.target docker.socket
Requires=docker.socket

[Service]
Type=notify
# the default is not to use systemd for cgroups because the delegate issues still
# exists and systemd currently does not support the cgroup feature set required
# for containers run by docker
ExecStart=/usr/bin/dockerd -H fd://
ExecReload=/bin/kill -s HUP $MAINPID

# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
# Uncomment TasksMax if your systemd version supports it.
# Only systemd 226 and above support this version.
TasksMax=infinity
TimeoutStartSec=0
# set delegate yes so that systemd does not reset the cgroups of docker containers
Delegate=yes
# kill only the docker process, not all processes in the cgroup
KillMode=process

[Install]
WantedBy=multi-user.target

修改的Dockerfile.aarch64文件

下面是修改过的Dockerfile.aarch64，查看原始文件：

# This file describes the standard way to build Docker on aarch64, using docker
#
# Usage:
#
# # Assemble the full dev environment. This is slow the first time.
# docker build -t docker -f Dockerfile.aarch64 .
#
# # Mount your source in an interactive container for quick testing:
# docker run -v `pwd`:/go/src/github.com/docker/docker --privileged -i -t docker bash
#
# # Run the test suite:
# docker run --privileged docker hack/make.sh test-unit test-integration-cli test-docker-py
#
# Note: AppArmor used to mess with privileged mode, but this is no longer
# the case. Therefore, you don't have to disable it anymore.
#

FROM aarch64/ubuntu:wily

# Packaged dependencies
RUN apt-get update && apt-get install -y \
	apparmor \
	aufs-tools \
	automake \
	bash-completion \
	btrfs-tools \
	build-essential \
	cmake \
	createrepo \
	curl \
	dpkg-sig \
	g++ \
	gcc \
	git \
	iptables \
	jq \
	libapparmor-dev \
	libc6-dev \
	libcap-dev \
	libltdl-dev \
	libsqlite3-dev \
	libsystemd-dev \
	mercurial \
	net-tools \
	parallel \
	pkg-config \
	python-dev \
	python-mock \
	python-pip \
	python-websocket \
	gccgo \
        golang \
	iproute2 \
	iputils-ping \
	vim-common \
	--no-install-recommends

# Get lvm2 source for compiling statically
ENV LVM2_VERSION 2.02.103
RUN mkdir -p /usr/local/lvm2 \
	&& curl -fsSL "https://mirrors.kernel.org/sourceware/lvm2/LVM2.${LVM2_VERSION}.tgz" \
		| tar -xzC /usr/local/lvm2 --strip-components=1
# See https://git.fedorahosted.org/cgit/lvm2.git/refs/tags for release tags

# Fix platform enablement in lvm2 to support aarch64 properly
RUN set -e \
	&& for f in config.guess config.sub; do \
		curl -fsSL -o "/usr/local/lvm2/autoconf/$f" "http://git.savannah.gnu.org/gitweb/?p=config.git;a=blob_plain;f=$f;hb=HEAD"; \
	done
# "arch.c:78:2: error: #error the arch code needs to know about your machine type"

# Compile and install lvm2
RUN cd /usr/local/lvm2 \
	&& ./configure \
		--build="$(gcc -print-multiarch)" \
		--enable-static_link \
	&& make device-mapper \
	&& make install_device-mapper
# See https://git.fedorahosted.org/cgit/lvm2.git/tree/INSTALL

# Install seccomp: the version shipped in trusty is too old
ENV SECCOMP_VERSION 2.3.1
RUN set -x \
	&& export SECCOMP_PATH="$(mktemp -d)" \
	&& curl -fsSL "https://github.com/seccomp/libseccomp/releases/download/v${SECCOMP_VERSION}/libseccomp-${SECCOMP_VERSION}.tar.gz" \
		| tar -xzC "$SECCOMP_PATH" --strip-components=1 \
	&& ( \
		cd "$SECCOMP_PATH" \
		&& ./configure --prefix=/usr/local \
		&& make \
		&& make install \
		&& ldconfig \
	) \
	&& rm -rf "$SECCOMP_PATH"

# Install Go
# We don't have official binary tarballs for ARM64, eigher for Go or bootstrap,
# so we use gccgo as bootstrap to build Go from source code.
# We don't use the official ARMv6 released binaries as a GOROOT_BOOTSTRAP, because
# not all ARM64 platforms support 32-bit mode. 32-bit mode is optional for ARMv8.
ENV GO_VERSION 1.7.4
RUN mkdir /usr/src/go && curl -x 192.168.1.11:9999 -fsSL https://golang.org/dl/go${GO_VERSION}.src.tar.gz | tar -v -C /usr/src/go -xz --strip-components=1 \
	&& cd /usr/src/go/src \
	&& GOOS=linux GOARCH=arm64 GOROOT_BOOTSTRAP="$(go env GOROOT)" ./make.bash

ENV PATH /usr/src/go/bin:$PATH
ENV GOPATH /go

# Only install one version of the registry, because old version which support
# schema1 manifests is not working on ARM64, we should skip integration-cli
# tests for schema1 manifests on ARM64.
ENV REGISTRY_COMMIT 47a064d4195a9b56133891bbb13620c3ac83a827
RUN set -x \
	&& export GOPATH="$(mktemp -d)" \
	&& git clone https://github.com/docker/distribution.git "$GOPATH/src/github.com/docker/distribution" \
	&& (cd "$GOPATH/src/github.com/docker/distribution" && git checkout -q "$REGISTRY_COMMIT") \
	&& GOPATH="$GOPATH/src/github.com/docker/distribution/Godeps/_workspace:$GOPATH" \
		go build -o /usr/local/bin/registry-v2 github.com/docker/distribution/cmd/registry \
	&& rm -rf "$GOPATH"

# Install notary and notary-server
ENV NOTARY_VERSION v0.4.2
RUN set -x \
	&& export GOPATH="$(mktemp -d)" \
	&& git clone https://github.com/docker/notary.git "$GOPATH/src/github.com/docker/notary" \
	&& (cd "$GOPATH/src/github.com/docker/notary" && git checkout -q "$NOTARY_VERSION") \
	&& GOPATH="$GOPATH/src/github.com/docker/notary/vendor:$GOPATH" \
		go build -o /usr/local/bin/notary-server github.com/docker/notary/cmd/notary-server \
	&& GOPATH="$GOPATH/src/github.com/docker/notary/vendor:$GOPATH" \
		go build -o /usr/local/bin/notary github.com/docker/notary/cmd/notary \
	&& rm -rf "$GOPATH"

# Get the "docker-py" source so we can run their integration tests
ENV DOCKER_PY_COMMIT e2655f658408f9ad1f62abdef3eb6ed43c0cf324
RUN git clone https://github.com/docker/docker-py.git /docker-py \
	&& cd /docker-py \
	&& git checkout -q $DOCKER_PY_COMMIT \
	&& pip install -r test-requirements.txt

# Set user.email so crosbymichael's in-container merge commits go smoothly
RUN git config --global user.email 'docker-dummy@example.com'

# Add an unprivileged user to be used for tests which need it
RUN groupadd -r docker
RUN useradd --create-home --gid docker unprivilegeduser

VOLUME /var/lib/docker
WORKDIR /go/src/github.com/docker/docker
ENV DOCKER_BUILDTAGS apparmor pkcs11 seccomp selinux

# Let us use a .bashrc file
RUN ln -sfv $PWD/.bashrc ~/.bashrc

# Register Docker's bash completion.
RUN ln -sv $PWD/contrib/completion/bash/docker /etc/bash_completion.d/docker

# Get useful and necessary Hub images so we can "docker load" locally instead of pulling
COPY contrib/download-frozen-image-v2.sh /go/src/github.com/docker/docker/contrib/
RUN ./contrib/download-frozen-image-v2.sh /docker-frozen-images \
	aarch64/buildpack-deps:jessie@sha256:6aa1d6910791b7ac78265fd0798e5abd6cb3f27ae992f6f960f6c303ec9535f2 \
	aarch64/busybox:latest@sha256:b23a6a37cf269dff6e46d2473b6e227afa42b037e6d23435f1d2bc40fc8c2828 \
	aarch64/debian:jessie@sha256:4be74a41a7c70ebe887b634b11ffe516cf4fcd56864a54941e56bb49883c3170 \
	aarch64/hello-world:latest@sha256:65a4a158587b307bb02db4de41b836addb0c35175bdc801367b1ac1ddeb9afda
# See also "hack/make/.ensure-frozen-images" (which needs to be updated any time this list is)

# Install tomlv, vndr, runc, containerd, tini, docker-proxy
# Please edit hack/dockerfile/install-binaries.sh to update them.
COPY hack/dockerfile/binaries-commits /tmp/binaries-commits
COPY hack/dockerfile/install-binaries.sh /tmp/install-binaries.sh
RUN /tmp/install-binaries.sh tomlv vndr runc containerd tini proxy

# Wrap all commands in the "docker-in-docker" script to allow nested containers
ENTRYPOINT ["hack/dind"]

# Upload docker source
COPY . /go/src/github.com/docker/docker

转载于:https://my.oschina.net/u/2306127/blog/803599