基于 Kubernetes 实现 CI/CD 配置

一、基本介绍

基于 Kubernetes 实现 CI/CD 配置，其实和往常那些 CI/CD 配置并没有太大区别。都是通过 提交代码，拉取代码，构建代码，发布代码来实现的。 只不过要是通过 K8s 来实现的话，则是需要将构建好的代码打包成镜像，通过镜像的方式来运行。

CI/CD 流程图：

开发将代码提交代码仓库后，我们便可以通过在 Jenkins 上配置脚本或是 Pipline 的方式来实现代码发布，其中发布有两种方式，一种是通过手动发布，另外一种可以通过 WebHook 插件来实现提交代码便自动发布（生产环境不建议自动发布）

• 脚本内容一般分为：克隆代码、编译代码、将编译好的代码打包成镜像、运行镜像几个步骤。

二、基于 Kubernetes 实现 CI/CD 配置

• 下面我们是通过容器的方式安装配置，物理安装可以看我前面的文章：传送门

1.配置 GitLab

1）安装 Docker-Compose

[root@k8s-master01 ~]# wget "https://github.com/docker/compose/releases/download/v2.3.2/docker-compose-$(uname -s)-$(uname -m)" -O /usr/local/bin/docker-compose	
[root@k8s-master01 ~]# chmod +x /usr/local/bin/docker-compose
[root@k8s-master01 ~]# docker-compose --version

2）安装 GitLab

[root@k8s-master01 ~]# vim docker-compose.yml
version: '3'
services:
  web:
    image: 'gitlab/gitlab-ce:14.8.5-ce.0'
    restart: always
    hostname: 192.168.1.1
    environment:
      GITLAB_OMNIBUS_CONFIG: |
        external_url 'http://192.168.1.1'
    ports:
      - '1080:80'
      - '1443:443'
      - '1022:22'
    volumes:
      - '/app/gitlab/config:/etc/gitlab'
      - '/app/gitlab/logs:/var/log/gitlab'
      - '/app/gitlab/data:/var/opt/gitlab'
[root@k8s-master01 ~]# docker-compose up -d

• 因为博主的电脑配置不是很高，所以就不使用上面的方式安装 GitLab，而是直接使用 GitHub 上面的仓库。

2.配置 Jenkins

1）安装 NFS 存储，并配置共享目录

[root@k8s-master01 ~]# yum -y install nfs-utils rpcbind
[root@k8s-master01 ~]# echo "/app/jenkins *(rw,sync,no_root_squash)" > /etc/exports
[root@k8s-master01 ~]# mkdir /app/jenkins
[root@k8s-master01 ~]# systemctl start rpcbind nfs

2）创建 PV 和 PVC

[root@k8s-master01 ~]# vim jenkins-pv.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
  name: jenkins-pv
spec:
  capacity:
    storage: 10Gi
  accessModes:
    - ReadWriteMany
  nfs:
    server: 192.168.1.1
    path: /app/jenkins
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: jenkins-pvc
spec:
  resources:
    requests:
      storage: 10Gi
  accessModes:
    - ReadWriteMany   
[root@k8s-master01 ~]# kubectl create -f jenkins-pv.yaml

2）创建 RBAC 授权

[root@k8s-master01 ~]# vim jenkins-sa.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: jenkins-sa
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: jenkins-cr
rules:
  - apiGroups: ["extensions","apps"]
    resources: ["deployments"]
    verbs: ["create","delete","get","list","watch","patch","update"]
  - apiGroups: [""]
    resources: ["services"]
    verbs: ["create","delete","get","list","watch","patch","update"]
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["create","delete","get","list","patch","update"]
  - apiGroups: [""]
    resources: ["pods/exec"]
    verbs: ["create","delete","get","list","patch","update"]
  - apiGroups: [""]
    resources: ["pods/log"]
    verbs: ["get","list","update"]
  - apiGroups: [""]
    resources: ["secrets"]
    verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: jenkins-crb
roleRef:
  kind: ClusterRole
  name: jenkins-cr
  apiGroup: rbac.authorization.k8s.io
subjects:
- kind: ServiceAccount
  name: jenkins-sa
  namespace: default
[root@k8s-master01 ~]# kubectl create -f jenkins-sa.yaml

3）创建 StatefulSet

[root@k8s-master01 ~]# vim jenkins-statefulset.yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: jenkins
spec:
  serviceName: jenkins
  replicas: 1
  selector:
    matchLabels:
      app: jenkins
  template:
    metadata:
      name: "jenkins"
      labels:
        app: jenkins
    spec:
      serviceAccountName: jenkins-sa
      containers:
      - name: jenkins
        image: jenkins/jenkins:lts
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 8080
        - containerPort: 50000
        volumeMounts:
        - name: jenkins
          mountPath: /var/jenkins_home
      volumes:
      - name: jenkins
        persistentVolumeClaim:
          claimName: jenkins-pvc          
[root@k8s-master01 ~]# chown -R 1000 /app/jenkins
[root@k8s-master01 ~]# kubectl create -f jenkins-statefulset.yaml

4）创建 Service

[root@k8s-master01 ~]# vim jenkins-svc.yaml
apiVersion: v1
kind: Service
metadata:
  name: jenkins
spec:
  type: NodePort
  ports:
  - name: http
    port: 8080
    targetPort: 8080
    nodePort: 30080
  - name: agent
    port: 50000
    targetPort: 50000
    nodePort: 30090
  selector:
    app: jenkins
[root@k8s-master01 ~]# kubectl create -f jenkins-svc.yaml

5）配置 Jenkins

[root@k8s-master01 ~]# cat /app/jenkins/secrets/initialAdminPassword 
a303d66e915e4ee5b26648a64fdff4be

http://192.168.1.1:30080/

我们这里安装推荐的插件即可，后面有需求可以再进行安装

3.实现 CI/CD 配置

1）在 Jenkins 宿主机上创建 SSH 密钥

[root@k8s-master01 ~]# ssh-keygen -t rsa								# 三连回车
[root@k8s-master01 ~]# cat ~/.ssh/id_rsa.pub							# 查看公钥

2）将公钥上传到 GitLab 上

3）将仓库克隆到本地

[root@k8s-master01 ~]# git clone git@github.com:ChenZhuang1217/test.git

4）编写 Go 代码

[root@k8s-master01 ~]# cd test
[root@k8s-master01 test]# vim main.go
package main
import (
	"fmt"
	"net/http"
)
func HelloHandler(w http.ResponseWriter, r *http.Request) {
	fmt.Fprintf(w, "Hello World")
}
func main() {
	http.HandleFunc("/", HelloHandler)
	http.ListenAndServe(":8080", nil)
}

5）编写 Dockerfile

[root@k8s-master01 test]# vim Dockerfile
FROM golang:1.16 as builder
ENV GO111MODULE=on \
    GOPROXY=https://goproxy.cn,direct
WORKDIR /app
COPY . .
RUN CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -ldflags="-w -s" -o main main.go

FROM busybox:1.28.4
WORKDIR /app
COPY --from=builder /app/ .
EXPOSE 8080
CMD ["./main"]
[root@k8s-master01 test]# docker build -t test-web-server:devops-$(date +%Y-%m-%d-%H-%M-%S) .

6）提交代码（Git 教程可以看博主前面写的文章：Git）

[root@k8s-master01 test]# git add .														# 提交到暂存区
[root@k8s-master01 test]# git config --global user.email "Zhuang_zz1217@163.com"		# 配置用户邮箱
[root@k8s-master01 test]# git commit -m "This is test CI/CD"							# 提交到本地仓库
[root@k8s-master01 test]# git push														# 推送到远程仓库

7）创建 Deployment 和 Service

[root@k8s-master01 ~]# vim test-web-server.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: test-web-server
spec:
  replicas: 1
  selector:
    matchLabels:
      app: test-web-server
  template:
    metadata:
      labels:
        app: test-web-server
    spec:
      containers:
      - name: test-web-server
        image: test-web-server:devops-2022-04-25-17-16-54
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
  name: test-web-server
spec:
  type: NodePort
  ports:
  - name: test-web-server
    port: 8080
    targetPort: 8080
    nodePort: 30188
  selector:
    app: test-web-server
[root@k8s-master01 ~]# kubectl create -f test-web-server.yaml

8）编写 Jenkins 发版脚本

[root@k8s-master01 ~]# vim test.sh
#!/bin/bash
# 固定时间格式
Second=$(date +%Y-%m-%d-%H-%M-%S)

# 备份旧的镜像
Image=$(kubectl -s https://192.168.1.1:6443 describe pod | grep Image: | awk '{print $2}' | grep test)
echo $Image > /opt/test-image-$Second

# 克隆代码
cd /root
if [ -d test ];
then
  mv test /opt/test-devops-$Second
  git clone git@github.com:ChenZhuang1217/test.git
else
  git clone git@github.com:ChenZhuang1217/test.git
fi

# 发布新的镜像
cd /root/test && docker build -t test-web-server:devops-$Second .

# 上传到镜像仓库
if [ $? -eq 0 ];
then
  docker tag test-web-server:devops-$Second harbor.tianya.com:5000/test-web-server:devops-$Second
  docker push harbor.tianya.com:5000/test-web-server:devops-$Second
else
  exit 1			# 退出 (防止运行下面命令)
fi

# 替换镜像
sed -i 's/image:.*/image: harbor.tianya.com:5000\/test-web-server:devops-'$Second'/g' /root/test-web-server.yaml

# 重启应用
kubectl delete -f /root/test-web-server.yaml
kubectl create -f /root/test-web-server.yaml
[root@k8s-master01 ~]# chmod +x test.sh

上面这个脚本有两步需要注意：

1. 上传到镜像仓库： 如果你们没有自己的镜像仓库，可以选择调整脚本或看博主前面写的文章来安装 Harbor 仓库。
2. 替换镜像： 我们上面配置的脚本是针对单个模块的，多个模块可以根据 for 循环来实现。

4.验证

1）在 Jenkins 上安装 SSH 插件

• 安装 SSH 插件的原因是因为，我们这个 Jenkins 是容器安装的，而脚本是在宿主机写的，所以通过远程到宿主机来运行脚本。

2）配置远程主机的用户名和密码

3）创建 Jenkins 私钥凭证（类型选择：SSH Username with private key）

4）配置 Jenkins 流水线

5）修改代码

6）在 Jenkins 上发布