目录
1.5 编译debian--hyperkube-base组件
一、环境准备
本次涉及到的相关组件及系统如下:
组件/os | 版本 | 其他 |
Docker-ce | 18.06 | |
Kubernetes | 1.13 | |
Golang | 1.12.12 | |
Os | NeoKylin Desktop B55.1 |
二、基础镜像
由k8s源码可知,源码是基于Debian系统构建的,因此准备好hub上龙芯的Debian源镜像
docker pull loongnix/debian:stretch
关于该镜像,没有集成golang,因此我们可以直接下载hub上龙芯已经集成的那个镜像,也可以自己基于此镜像手动集成,这里我们选择后者,集成方法如下:
docker run -it loongnix/debian:stretch /bin/bash
echo "deb http://mirrors.ustc.edu.cn/debian stable main contrib non-free" > /etc/apt/sources.list
apt-get udate
apt-get install golang-go
go version #查看golang版本
docker commit [CONTAINER ID] debian-golang:1.11.6
注意:docker官方说明,建议不要用commit生成镜像,这里可以自己写个Dockerfile。
三、源码编译
1.1 下载源码
git clone -b release-1.13 https://gitee.com/leechm/kubernetes.git
1.2 编译kube-cross组件
目录:kubernetes/build/build-image/cross/
准备好etcd和protobuf,下载对应的版本,在Dockerfile里面可以看到。
git clone -b v3.2.24 https://gitee.com/leechm/etcd.git
git clone -b v3.0.0 https://gitee.com/leechm/protobuf.git
本地编译上述组件
首先编译etcd,进入etcd中,修改etcdmain/etcd.go,
执行./build,完成之后在当前bin目录下生成可执行程序。
编译protobuf,进入protobuf中,步骤如下:
./autogen.sh
./configure --prefix=/此安装目录自己命名
make
make install
cd /此安装目录自己命名
在这个目录下有三个文件夹 bin include lib 这个就是拿来写Dockerfile的
注意:上述两个组件都有可执行的二进制程序生成,在本地运行一下,未报错即可使用
接着准备好tools,xerrors和mod组件
git clone -b release-branch.go1.11 https://gitee.com/leechm/golang-tools.git
git clone https://gitee.com/leechm/xerrors.git
git clone https://gitee.com/leechm/mod.git
完成上述两个组件的编译之后。进入kubernetes/build/build-image/cross/下,修改Makefile,删除--pull,修改Dockerfile,重点在于etcd和protobuf这两个组件的写法。还有golang-tools,xerrors和mod组件这三个组件在镜像中的位置:放到$GOPATH/src/golang.org/x/下,例如:
最后执行make build即可,编译过程中报错什么,解决一下即可,若为未安装某包,替换源码包仓库即可,这里我们提供以下三个仓库
deb http://mirrors.huaweicloud.com/debian stretch main
deb http://archive.ubuntu.com/ubuntu xenial main universe
deb http://mirrors.ustc.edu.cn/debian stable main contrib non-free
编译完成,docker images查看生成的镜像。
1.3 编译pause组件
目录:kubernetes/build/pause
修改Makefile,删除--pull和增加mips64le相关代码,执行make pause即可生成pause可执行文件。
(若报错:/usr/bin/ld: cannot find -lc,装上glibc-static即可)
最后修改Dockerfile,执行docker build -t pause:tag . 即可。
1.4 编译debian-base组件
目录:kubernetes/build/debian-base
修改Makefile,删除--pull选项,增加mips64le架构代码。
注意:在GitHub上,qemu-mips64-static.tar.gz 这个组件已经提供了。
我们只需要执行make build即可。
1.5 编译debian--hyperkube-base组件
目录:kubernetes/build/debian-hyperkube-base/
修改Makefile,删除--pull选项,增加mips64le架构代码。由Makefile可知,需要network-plugins插件
git clone -b v0.7.5 https://gitee.com/leechm/plugins.git
执行./build.sh
即可在./bin/生成二进制文件
cd /bin
tar czvf cni-plugins-mips64-v0.7.5.tgz *
最后执行make build 即可。
1.6 编译debian-iptables组件
目录:kubernetes/build/debian-iptables/
修改Makefile,删除--pull选项,增加mips64le架构代码。执行make build即可。。
上述步骤完成之后,docker images查看相关base镜像的脚本安装包如下:
1.7 编译k8s组件
依赖的镜像上面已经编译好了,接下里需要修改kubernetes/build/lib/release.sh这个文件,删除--pull这个参数选项。
修改hack/lib/golang.sh,在KUBE_SERVER_PLATFORMS,KUBE_CLIENT_PLATFORMS和KUBE_TEST_PLATFORMS中增加linux/mip64le,也可以去掉除linux/mips64le以外的其他平台,以此来减少编译所用时间。
上述就是增加对mips架构的支持,详细的修改见附录1
配置好mips支持架构之后,执行如下命令:
KUBE_BUILD_PLATFORMS=linux/mips64le KUBE_BUILD_CONFORMANCE=n KUBE_BUILD_HYPERKUBE=n make release-images GOFLAGS=-v GOGCFLAGS="-N -l"
参数说明:
UBE_BUILD_CONFORMANCE=n 和 KUBE_BUILD_HYPERKUBE=n 参数配置是否构建 hyperkube-mips64le 和 conformance-mips64le 镜像,默认是 y 构建,这里设置为 n 暂时不需要构建了。
make release-images 表示执行编译并生成镜像 的tar 包。
GOFLAGS=-v开启verbose日志,GOGCFLAGS=”-N -l”禁止编译优化和内联,减小可执行程序大小。
分析:若出现错误,无法根据报错提示判断什么错误,可以使用docker logs [CONTAINER ID]查看容器报错信息。
最后编译好的kubernetes 组件 docker 镜像以 tar 包的形式存在 kubernetes/_output/release-tars/mips64le 目录中。执行docker load -i 即可导入本地。
生成的核心组件二进制可执行文件以及镜像,在 _output/release-stage/server/linux-mips64le/kubernetes/server/bin/
四、其他组件
1.1 Coredns编译
下载源码
git clone -b v1.2.2 https://gitee.com/leechm/coredns.git
源码存放路径为:$GOPATH/src/github.com/coredns/
cd到该路径下,执行go build 命令,一般会报cannot find package的错误,这个只需要把相关缺少的组件放到对应的位置即可,此时我们开启golang代理模式,根据go.mod文件自动联网下载相关依赖
# 启用 Go Modules 功能
export GO111MODULE=on
# 配置 GOPROXY 环境变量
export GOPROXY=https://goproxy.io
最后编译出来的二进制文件验证如下:
然后根据目录中的Dockerfile,即可生成coredns镜像。
1.2 Flannel编译
下载源码:
git clone -b v0.10.0 https://gitee.com/leechm/flannel.git
源码存放路径为:$GOPATH/src/github.com/coreos/
官方提供的方案是基于alpine镜像做的,有兴趣的可以研究一下这个方法,在龙芯开源社区上也提供了mips架构的相关包,如:http://doc.loongnix.org/web/#/78?page_id=293
这里我们基于中标麒麟base镜像
根据架构修改Makefile,注意CGO_ENABLED的值为1,cp一份Dockerfile.arm64为Dockerfile.mips64le,修改Dockerfile.mips64le.注意若是基于centos,则GOPACH路径需要修改,如下:
最后执行make image即可。
1.3 Etcd编译
下载源码:
git clone -b v3.2.24 https://gitee.com/leechm/etcd.git
下载龙芯alpine镜像
docker pull loongnix/alpine:3.11
修改etcdmain/etcd.go
然后执行./build,即可在./bin/目录下生成可执行程序。
验证生成文件:
修改Dockerfile-release,接着执行:docker build -f Dockerfile-release -t etcd:v3.2.24 .
1.4 registry2.7.1镜像
mkdir -p $GOPATH/src/github.com/docker/
cd $GOPATH/src/github.com/docker/
git clone -b v2.7.1 https://gitee.com/leechm/distribution.git
make binaries
Dockerfile如下:
执行docker build 即可生成镜像。
验证一下:
docker run -d -p 5000:5000 --restart always --name registry registry:2.7.1
附录1
diff -urN kubernetes-1.13/build/common.sh kubernetes/build/common.sh
--- kubernetes-1.13/build/common.sh 2020-07-06 19:56:02.000000000 +0800
+++ kubernetes/build/common.sh2020-07-11 16:58:13.000000000 +0800
@@ -59,9 +59,6 @@
# limitations under the License.
# Common utilities, variables and checks for all build scripts.
- export PS4='+${BASH_SOURCE}:${LINENO}: '
- set -x
set -o errexit
set -o nounset
set -o pipefail
@@ -94,11 +96,11 @@ kube::build::get_docker_wrapped_binaries
### If you change any of these lists, please also update DOCKERIZED_BINARIES
### in build/BUILD. And kube::golang::server_image_targets
local targets=(
- cloud-controller-manager,"loongnixk8s/debian-base-${arch}:${debian_base_version}"
- kube-apiserver,"loongnixk8s/debian-base-${arch}:${debian_base_version}"
- kube-controller-manager,"loongnixk8s/debian-base-${arch}:${debian_base_version}"
- kube-scheduler,"loongnixk8s/debian-base-${arch}:${debian_base_version}"
- kube-proxy,"loongnixk8s/debian-iptables-${arch}:${debian_iptables_version}"
+ cloud-controller-manager,"k8s.gcr.io/debian-base-${arch}:${debian_base_version}"
+ kube-apiserver,"k8s.gcr.io/debian-base-${arch}:${debian_base_version}"
+ kube-controller-manager,"k8s.gcr.io/debian-base-${arch}:${debian_base_version}"
+ kube-scheduler,"k8s.gcr.io/debian-base-${arch}:${debian_base_version}"
+ kube-proxy,"k8s.gcr.io/debian-iptables-${arch}:${debian_iptables_version}"
)
echo "${targets[@]}"
@@ -529,6 +537,7 @@ function kube::build::ensure_data_container() {
--volume /usr/local/go/pkg/linux_arm_cgo
--volume /usr/local/go/pkg/linux_arm64_cgo
--volume /usr/local/go/pkg/linux_ppc64le_cgo
- --volume /usr/local/go/pkg/linux_mips64le_cgo
--volume /usr/local/go/pkg/darwin_amd64_cgo
--volume /usr/local/go/pkg/darwin_386_cgo
--volume /usr/local/go/pkg/windows_amd64_cgo
diff -urN kubernetes-1.13/cluster/clientbin.sh kubernetes/cluster/clientbin.sh
--- kubernetes-1.13/cluster/clientbin.sh 2020-07-06 19:56:02.000000000 +0800
+++ kubernetes/cluster/clientbin.sh 2020-07-11 16:58:13.000000000 +0800
@@ -59,9 +59,6 @@
ppc64le*)
host_arch=ppc64le
;;
- mips64*)
- host_arch=mips64le
- ;;
*)
echo "Unsupported host arch. Must be x86_64, 386, arm, s390x or ppc64le." >&2
exit 1
diff -urN kubernetes-1.13/cluster/images/conformance/Makefile kubernetes/cluster/images/conformance/Makefile
--- kubernetes-1.13/cluster/images/conformance/Makefile 2020-07-06 19:57:12.000000000 +0800
+++ kubernetes/cluster/images/conformance/Makefile 2020-07-11 16:58:13.000000000 +0800
@@ -25,7 +25,7 @@
E2E_TEST_BIN?=$(shell pwd)/../../../$(OUT_DIR)/dockerized/bin/linux/$(ARCH)/e2e.test
CLUSTER_DIR?=$(shell pwd)/../../../cluster/
-BASEIMAGE=staging-k8s.gcr.io/debian-hyperkube-base-mips64:0.12.0
+BASEIMAGE=k8s.gcr.io/debian-hyperkube-base-$(ARCH):0.12.0
TEMP_DIR:=$(shell mktemp -d -t conformanceXXXXXX)
all: build
@@ -48,7 +48,7 @@
cd ${TEMP_DIR} && sed -i.back "s|BASEIMAGE|${BASEIMAGE}|g" Dockerfile
- docker build -t ${REGISTRY}/conformance-${ARCH}:${VERSION} ${TEMP_DIR}
+ docker build --pull -t ${REGISTRY}/conformance-${ARCH}:${VERSION} ${TEMP_DIR}
rm -rf "${TEMP_DIR}"
push: build
diff -urN kubernetes-1.13/cluster/images/hyperkube/Makefile kubernetes/cluster/images/hyperkube/Makefile
--- kubernetes-1.13/cluster/images/hyperkube/Makefile 2020-07-06 19:58:00.000000000 +0800
+++ kubernetes/cluster/images/hyperkube/Makefile 2020-07-11 16:58:13.000000000 +0800
@@ -22,7 +22,7 @@
OUT_DIR?=_output
HYPERKUBE_BIN?=$(shell pwd)/../../../$(OUT_DIR)/dockerized/bin/linux/$(ARCH)/hyperkube
-BASEIMAGE=staging-k8s.gcr.io/debian-hyperkube-base-mips64:0.12.0
+BASEIMAGE=k8s.gcr.io/debian-hyperkube-base-$(ARCH):0.12.0
TEMP_DIR:=$(shell mktemp -d -t hyperkubeXXXXXX)
all: build
@@ -39,7 +39,7 @@
cd ${TEMP_DIR} && sed -i.back "s|BASEIMAGE|${BASEIMAGE}|g" Dockerfile
- docker build -t ${REGISTRY}/hyperkube-${ARCH}:${VERSION} ${TEMP_DIR}
+ docker build --pull -t ${REGISTRY}/hyperkube-${ARCH}:${VERSION} ${TEMP_DIR}
rm -rf "${TEMP_DIR}"
push: build
diff -urN kubernetes-1.13/hack/lib/golang.sh kubernetes/hack/lib/golang.sh
--- kubernetes-1.13/hack/lib/golang.sh 2020-07-06 20:02:37.000000000 +0800
+++ kubernetes/hack/lib/golang.sh 2020-07-11 16:58:14.000000000 +0800
@@ -118,7 +118,6 @@
readonly KUBE_SERVER_PLATFORMS=(
linux/amd64
linux/arm
- linux/mips64le
linux/arm64
linux/s390x
linux/ppc64le
@@ -129,7 +128,6 @@
linux/amd64
linux/arm
linux/arm64
- linux/mips64le
linux/s390x
linux/ppc64le
windows/amd64
@@ -141,7 +139,6 @@
linux/386
linux/arm
linux/arm64
- linux/mips64le
linux/s390x
linux/ppc64le
darwin/amd64
@@ -156,7 +153,6 @@
linux/arm
linux/arm64
linux/s390x
- linux/mips64le
linux/ppc64le
darwin/amd64
windows/amd64
@@ -334,10 +330,6 @@
export CGO_ENABLED=1
export CC=s390x-linux-gnu-gcc
;;
- "linux/mips64le")
- export CGO_ENABLED=1
- export CC=mips64el-linux-gnu-gcc
- ;;
esac
fi
}
diff -urN kubernetes-1.13/hack/lib/util.sh kubernetes/hack/lib/util.sh
--- kubernetes-1.13/hack/lib/util.sh 2020-07-06 19:49:11.000000000 +0800
+++ kubernetes/hack/lib/util.sh 2020-07-11 16:58:14.000000000 +0800
@@ -145,11 +145,8 @@
ppc64le*)
host_arch=ppc64le
;;
- mips64*)
- host_arch=mips64le
- ;;
*)
- kube::log::error "Unsupported host arch. Must be x86_64, mips64le, 386, arm, arm64, s390x or ppc64le."
+ kube::log::error "Unsupported host arch. Must be x86_64, 386, arm, arm64, s390x or ppc64le."
exit 1
;;
esac
diff -urN kubernetes-1.13/hack/local-up-cluster.sh kubernetes/hack/local-up-cluster.sh
--- kubernetes-1.13/hack/local-up-cluster.sh 2020-07-06 19:48:14.000000000 +0800
+++ kubernetes/hack/local-up-cluster.sh 2020-07-11 16:58:14.000000000 +0800
@@ -326,11 +326,8 @@
ppc64le*)
host_arch=ppc64le
;;
- mips64*)
- host_arch=mips64le
- ;;
*)
- echo "Unsupported host arch. Must be x86_64, mips64el, 386, arm, arm64, s390x or ppc64le." >&2
+ echo "Unsupported host arch. Must be x86_64, 386, arm, arm64, s390x or ppc64le." >&2
exit 1
;;
esac
diff -urN kubernetes-1.13/test/images/image-util.sh kubernetes/test/images/image-util.sh
--- kubernetes-1.13/test/images/image-util.sh 2020-07-06 20:03:19.000000000 +0800
+++ kubernetes/test/images/image-util.sh 2020-07-11 16:58:16.000000000 +0800
@@ -25,7 +25,7 @@
source "${KUBE_ROOT}/hack/lib/util.sh"
# Mapping of go ARCH to actual architectures shipped part of multiarch/qemu-user-static project
-declare -A QEMUARCHS=( ["amd64"]="x86_64" ["arm"]="arm" ["arm64"]="aarch64" ["ppc64le"]="ppc64le" ["s390x"]="s390x" ["mips64le"]="mips64")
+declare -A QEMUARCHS=( ["amd64"]="x86_64" ["arm"]="arm" ["arm64"]="aarch64" ["ppc64le"]="ppc64le" ["s390x"]="s390x" )
# Returns list of all supported architectures from BASEIMAGE file
listArchs() {
diff -urN kubernetes-1.13/vendor/github.com/google/cadvisor/fs/fs.go kubernetes/vendor/github.com/google/cadvisor/fs/fs.go
--- kubernetes-1.13/vendor/github.com/google/cadvisor/fs/fs.go 2020-07-06 20:06:34.000000000 +0800
+++ kubernetes/vendor/github.com/google/cadvisor/fs/fs.go 2020-07-11 16:58:17.000000000 +0800
@@ -524,8 +524,8 @@
return nil, fmt.Errorf("stat failed on %s with error: %s", dir, err)
}
- major := major(uint64(buf.Dev))
- minor := minor(uint64(buf.Dev))
+ major := major(buf.Dev)
+ minor := minor(buf.Dev)
for device, partition := range self.partitions {
if partition.major == major && partition.minor == minor {
return &DeviceInfo{device, major, minor}, nil
diff -urN kubernetes-1.13/vendor/github.com/vishvananda/netns/netns_linux.go kubernetes/vendor/github.com/vishvananda/netns/netns_linux.go
--- kubernetes-1.13/vendor/github.com/vishvananda/netns/netns_linux.go 2020-07-06 20:07:37.000000000 +0800
+++ kubernetes/vendor/github.com/vishvananda/netns/netns_linux.go 2020-07-11 16:58:17.000000000 +0800
@@ -21,7 +21,6 @@
"arm": 375,
"mips": 4344,
"mipsle": 4344,
- "mips64le": 5303,
"ppc64": 350,
"ppc64le": 350,
"s390x": 339,
diff -urN kubernetes-1.13/vendor/golang.org/x/sys/unix/zsyscall_linux_mips64le.go kubernetes/vendor/golang.org/x/sys/unix/zsyscall_linux_mips64le.go
--- kubernetes-1.13/vendor/golang.org/x/sys/unix/zsyscall_linux_mips64le.go 2020-07-06 20:08:36.000000000 +0800
+++ kubernetes/vendor/golang.org/x/sys/unix/zsyscall_linux_mips64le.go 2020-07-11 16:58:17.000000000 +0800
@@ -57,16 +57,6 @@
return
}
- // THIS FILE IS GENERATED BY THE COMMAND AT THE TOP; DO NOT EDIT
-func InotifyInit() (fd int, err error) {
- r0, _, e1 := RawSyscall(SYS_INOTIFY_INIT, 0, 0, 0)
- fd = int(r0)
- if e1 != 0 {
- err = errnoErr(e1)
- }
- return
-}
-
// THIS FILE IS GENERATED BY THE COMMAND AT THE TOP; DO NOT EDIT
func openat(dirfd int, path string, flags int, mode uint32) (fd int, err error) {