如何在本地搭建k8s开发环境

一、Hardware & OS

Kubernetes is a large project, and compiling it can use a lot of resources. We recommend the following for any physical or virtual machine being used for building Kubernetes.

• 8GB of RAM
• 50GB of free disk space

操作系统选择 centos 7

k8s选择目前最新 release-1.24 版本

二、基础软件安装

2.1 GNU Development Tools

sudo yum groupinstall "Development Tools"

2.2 container-runtime

This page provides an outline of how to use several common container runtimes with Kubernetes.

• containerd
• CRI-O
• Docker Engine
• Mirantis Container Runtime

可选上述四种之一，这里以docker为例。

docker

先卸载旧版本

 sudo yum remove docker \
                  docker-client \
                  docker-client-latest \
                  docker-common \
                  docker-latest \
                  docker-latest-logrotate \
                  docker-logrotate \
                  docker-engine

安装新版本

sudo yum install -y yum-utils
sudo yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo    
sudo yum install docker-ce docker-ce-cli containerd.io docker-compose-plugin

修改配置文件 /etc/docker/daemon.json

{
    "registry-mirrors": ["http://hub-mirror.c.163.com"],
    "exec-opts": ["native.cgroupdriver=systemd"]
}

cri-dockerd

下载源码

git clone git@github.com:Mirantis/cri-dockerd.git

To build this code (in a POSIX environment):

mkdir bin
cd src && go get && go build -o ../bin/cri-dockerd

To install, on a Linux system that uses systemd, and already has Docker Engine installed

# Run these commands as root
mkdir -p /usr/local/bin
install -o root -g root -m 0755 bin/cri-dockerd /usr/local/bin/cri-dockerd
cp -a packaging/systemd/* /etc/systemd/system
sed -i -e 's,/usr/bin/cri-dockerd,/usr/local/bin/cri-dockerd,' /etc/systemd/system/cri-docker.service
systemctl daemon-reload
systemctl enable cri-docker.service
systemctl enable --now cri-docker.socket

For cri-dockerd, the CRI socket is /run/cri-dockerd.sock by default.

2.3 Go

下载二进制程序：https://go.dev/dl/go1.18.1.linux-amd64.tar.gz

rm -rf /usr/local/go && tar -C /usr/local -xzf go1.18.1.linux-amd64.tar.gz
export PATH=$PATH:/usr/local/go/bin
​
# 确认安装成功
go version 

2.4 PyYAML

yum install python3
python3 -m pip install --upgrade pip setuptools wheel -i https://mirrors.aliyun.com/pypi/simple/
pip3 config set global.index-url https://pypi.tuna.tsinghua.edu.cn/simple
pip install pyyaml

2.5 CFSSL

git clone git@github.com:cloudflare/cfssl.git
cd cfssl
make
export PATH=$PATH:`pwd`/bin

三、编译k8s及相关组件

1、下载k8s

在github上将k8s项目fork一份到个人项目，下载自己仓库的k8s，并使用当前release的最新版本。

2、安装etcd

./hack/install-etcd.sh
# 根据上面安装后的结果提示配置etcd bin路径到PATH，如下面格式
export PATH="$GOPATH/src/k8s.io/kubernetes/third_party/etcd:${PATH}"

3、编译k8s

# 可指定编译某个模块，如 make WHAT=cmd/<subsystem>
make WHAT=cmd/kubectl

四、启动本地集群

进入kubernetes源码目录，执行命令

# 选择docker作为container-runtime，执行以下命令
export CONTAINER_RUNTIME_ENDPOINT="unix:///run/cri-dockerd.sock"
​
# 启动本地集群，后续启动如果不需要编译可加参数-O
./hack/local-up-cluster.sh 

五、使用K8S集群

5.1 通过命令运行pod

为了方便使用kubectl.sh，在 /etc/profile中添加cluster路径到PATH中，进入k8s目录

export PATH=$PATH:`pwd`/cluster

运行nginx

kubectl.sh run nginx --image=nginx --port=80

查看pods运行状态

kubectl.sh get pods

查看pod详情（pod能运行失败，可以通过该命令查看错误原因）

kubectl.sh describe pods nginx

删除pod

kubectl.sh delete pods nginx

进入docker的shell界面

kubectl.sh exec -it nginx /bin/bash

常见问题

1、Get "https://k8s.gcr.io/v2/": dial tcp 64.233.189.82:443: i/o timeout

pulling image "http://k8s.gcr.io/pause:3.1" 失败，可以通过以下命令解决，其他镜像处理方法类似

docker pull mirrorgooglecontainers/pause:3.1
docker tag docker.io/mirrorgooglecontainers/pause:3.1 k8s.gcr.io/pause:3.1

5.2 通过yaml配置文件运行pod

1、创建deployment文件并部署：nginx-deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  selector:
    matchLabels:
      app: nginx
  replicas: 3
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.7.9
        imagePullPolicy: IfNotPresent # Always/IfNotPresent/Never
        ports:
        - containerPort: 80
kubectl.sh apply -f nginx-deployment.yaml

2、创建service文件并部署：nginx-service.yaml

apiVersion: v1
kind: Service
metadata:
  name: nginx-service
  labels:
    app: nginx-service
spec:
  type: NodePort
  selector:
    app: nginx
  ports:
  - port: 8000
    targetPort: 80
    nodePort: 30080
kubectl.sh apply -f nginx-service.yaml

3、查看当前运行的pods、deployment、service

kubectl.sh get pods
kubectl.sh get deployment
kubectl.sh get svc

4、 访问网页效果

参考资料

https://github.com/kubernetes/community/blob/master/contributors/devel/development.md

https://github.com/kubernetes/community/blob/master/contributors/devel/running-locally.md

https://kubernetes.io/docs/setup/production-environment/container-runtimes/#docker

https://docs.docker.com/engine/install/centos/

https://github.com/Mirantis/cri-dockerd

https://golang.org/doc/install

https://github.com/kubernetes/community/blob/master/contributors/guide/github-workflow.md

https://etcd.io/

https://github.com/cloudflare/c