6. 业务系统基于K8S的CICD

将业务系统迁移到k8s中运行还不够，还应该让它能够持续化的集成部署，也就是CI/CD的过程，该过程必须保证自动化。

使用k8s进行CI/CD有以下优点：

1. 环境稳定

2. 服务不中断

3. 一次构建，多环境运行

本文通过K8S + Gitlab + Gitlab CI实现k8s集群的自动化集成部署。

环境如下：

ip	role	hostname	domain
192.168.1.51	k8s master1	master1	none
192.168.1.52	k8s master2	master2	none
192.168.1.53	k8s master3	master3	none
192.168.1.54	k8s node1	node1	none
192.168.1.55	k8s node2	node2	none
192.168.1.56	k8s node3	node3	none
192.168.1.57	gitlab server	gitlab	gitlab.lzxlinux.cn
192.168.1.59	harbor server	harbor1	hub.lzxlinux.cn
192.168.1.60	harbor server	harbor2	harbor.lzxlinux.cn

---

搭建Gitlab服务器

搭建Gitlab服务器，参考这里：Docker + Gitlab + Gitlab CI（一），此处省略搭建过程。

没有指定节点时，默认操作在所有节点进行。另外请清空harbor仓库hub.lzxlinux.cn中kubernetes项目的所有镜像，仅保留Dockerfile中需要的基础镜像。

• 添加本地dns：

# echo '54.153.54.194 packages.gitlab.com' >> /etc/hosts

# echo '192.168.1.57 gitlab.lzxlinux.cn' >> /etc/hosts

• 注册runner：

k8s集群master节点需要注册runner，以及harbor服务器。runner是shell类型，下面以master1节点为例，

# curl -L https://packages.gitlab.com/install/repositories/runner/gitlab-ci-multi-runner/script.rpm.sh | sudo bash

# yum install -y gitlab-ci-multi-runner

# gitlab-ci-multi-runner status
gitlab-runner: Service is running!

# usermod -aG docker gitlab-runner

# systemctl restart docker

# gitlab-ci-multi-runner restart

# mkdir -p /etc/gitlab/ssl

# scp [email protected]:/etc/gitlab/ssl/gitlab.lzxlinux.cn.crt /etc/gitlab/ssl

# gitlab-ci-multi-runner register \
  --tls-ca-file=/etc/gitlab/ssl/gitlab.lzxlinux.cn.crt \
  --url "https://gitlab.lzxlinux.cn/" \
  --registration-token "4kr9ZmLMWasYxqB2tSzQ" \
  --name "master1" \
  --tag-list "dev, master1" \
  --run-untagged="false" \
  --locked="false" \
  --executor "shell"
  
# gitlab-ci-multi-runner list
Listing configured runners                          ConfigFile=/etc/gitlab-runner/config.toml
master1                                             Executor=shell Token=1ihYEgDcnaqqXuiygHhn URL=https://gitlab.lzxlinux.cn/

• 给予权限：

要在k8s集群部署项目，需要以root身份执行命令kubectl apply -f filename.yaml。

因此3个master节点需要给gitlab-runner用户以root用户执行这条命令的权限。

# visudo

gitlab-runner   ALL=(ALL) NOPASSWD: ALL

• 创建项目：

新建一个群组kubernetes，

在该群组下新建4个项目：cronjob-demo、springboot-web-demo、debbo-demo、web-demo，

• push代码到gitlab（harbor1）：

以cronjob-demo项目为例，

# cd /software

# git config --global user.name "admin"

# git config --global user.email "[email protected]"

# git -c http.sslVerify=false clone https://gitlab.lzxlinux.cn/kubernetes/cronjob-demo.git

# mv mooc-k8s-demo-docker/cronjob-demo/* cronjob-demo/

# cp mooc-k8s-demo-docker/README.md cronjob-demo/

# cd cronjob-demo/

# tree -L 2
.
├── cronjob.yaml
├── Dockerfile
├── pom.xml
├── README.md
└── src
    └── main

2 directories, 4 files

# git add .

# git commit -m 'first commit'

# git -c http.sslVerify=false push origin master

其余项目与上面类似操作，将项目代码准备完毕。

• 创建harbor仓库的secret（master1）：

# kubectl create secret docker-registry hub-secret --docker-server=hub.lzxlinux.cn --docker-username=admin --docker-password=Harbor12345
secret/hub-secret created

# kubectl create secret docker-registry harbor-secret --docker-server=harbor.lzxlinux.cn --docker-username=admin --docker-password=Harbor12345
secret/harbor-secret created

# kubectl get secret

NAME                  TYPE                                  DATA   AGE
default-token-f758k   kubernetes.io/service-account-token   3      9d
harbor-secret         kubernetes.io/dockerconfigjson        1      10s
hub-secret            kubernetes.io/dockerconfigjson        1      21s

这样后续就可以在部署项目的yaml文件中使用secret来登录harbor仓库了。

• 所有项目设置：

设置 → 仓库 → Protected Branches，设置No one Allow to push到master分支。

当项目有代码风格检查及单元测试时，设置 → 通用 → 合并请求，勾选流水线必须成功，这里我们不设置这个。

接下来分别演示项目在 测试 → 预发布 → 生产 等环境下的k8s中的部署。

当然，我们这里只有一个k8s集群，所以项目实际上部署在一个k8s中，通过3个master节点代表3种环境分别部署项目。

---

Java定时任务的CI/CD

• 设置项目变量：

设置 → CI/CD → 变量，

• 修改代码（harbor1）：

创建新的dev分支，克隆代码到本地主机。

# cd /home

# git config --global user.name "admin"

# git config --global user.email "[email protected]"

# git -c http.sslVerify=false clone https://gitlab.lzxlinux.cn/kubernetes/cronjob-demo.git

# cd cronjob-demo/

# git checkout dev

# ls
cronjob.yaml  Dockerfile  pom.xml  README.md  src

# vim src/main/java/com/mooc/demo/cronjob/Main.java

package com.mooc.demo.cronjob;

import java.util.Random;

/**
 * Created by Michael on 2018/9/29.
 */
public class Main {
   

    public static void main(String args[]) {
   

        Random r = new Random();
        int time = r.nextInt(20)+10;
        System.out.println("I will working for "+time+" seconds! It's very good!");

        try{
   
            Thread.sleep(time*1000);
        }catch (Exception e) {
   
            e.printStackTrace();
        }
        System.out.println("All work is done! This is the first k8s ci/cd!");
    }
}

# vim cronjob.yaml

apiVersion: batch/v1beta1
kind: CronJob
metadata:
  name: cronjob-demo
spec:
  schedule: "*/1 * * * *"
  successfulJobsHistoryLimit: 3
  suspend: false
  concurrencyPolicy: Forbid
  failedJobsHistoryLimit: 1
  jobTemplate:
    spec:
      template:
        metadata:
          labels:
            app: cronjob-demo
        spec:
          restartPolicy: Never
          containers:
          - name: cronjob-demo
            image: hub.lzxlinux.cn/kubernetes/cronjob:latest
            imagePullPolicy: Always
          imagePullSecrets:
          - name: hub-secret

# vim .gitlab-ci.yml

stages:
  - login
  - build
  - dev
  - release
  - prod

docker-login:
  stage: login
  script:
    - docker login hub.lzxlinux.cn -u $HARBOR_USER -p $HARBOR_PASS
    - docker image prune -f
  tags:
    - harbor1
  only:
    - master

docker-build1:
  stage: build
  script:
    - mvn clean package
    - docker build -t hub.lzxlinux.cn/kubernetes/cronjob:latest .
    - docker push hub.lzxlinux.cn/kubernetes/cronjob:latest
  tags:
    - harbor1
  only:
    - master

docker-build2:
  stage: build
  script:
    - mvn clean package
    - docker build -t hub.lzxlinux.cn/kubernetes/cronjob:$TAG .
    - docker push hub.lzxlinux.cn/kubernetes/cronjob:$TAG
  when: manual
  tags:
    - harbor1
  only:
    - master

docker-build:
  stage: build
  script:
    - mvn clean package
    - docker build -t hub.lzxlinux.cn/kubernetes/cronjob:$CI_COMMIT_TAG .
    - docker push hub.lzxlinux.cn/kubernetes/cronjob:$CI_COMMIT_TAG
  tags:
    - harbor1
  only:
    - tags

k8s-dev:
  stage: dev
  script:
    - sudo su root -c "kubectl apply -f cronjob.yaml"
  tags:
    - dev
  only:
    - master
    
k8s-release:
  stage: release
  script:
    - cp -f cronjob.yaml cronjob-$TAG.yaml
    - sed -i "s#cronjob-demo#cronjob-$TAG#g" cronjob-$TAG.yaml
    - sed -i "s#hub.lzxlinux.cn/kubernetes/cronjob:latest#hub.lzxlinux.cn/kubernetes/cronjob:$TAG#g" cronjob-$TAG.yaml
    - sudo su root -c "kubectl apply -f cronjob-$TAG.yaml"
  after_script:
    - rm -f cronjob-$TAG.yaml
  when: manual
  tags:
    - release
  only:
    - master

k8s-prod:
  stage: prod
  script:
    - cp -f cronjob.yaml cronjob-$CI_COMMIT_TAG.yaml
    - VER=$(echo $CI_COMMIT_TAG |awk -F '.' '{
   print $1}')
    - sed -i "s#cronjob-demo#cronjob-$VER#g" cronjob-$CI_COMMIT_TAG.yaml
    - sed -i "s#hub.lzxlinux.cn/kubernetes/cronjob:latest#hub.lzxlinux.cn/kubernetes/cronjob:$CI_COMMIT_TAG#g" cronjob-$CI_COMMIT_TAG.yaml
    - sudo su root -c "kubectl apply -f cronjob-$CI_COMMIT_TAG.yaml"
  after_script:
    - rm -f cronjob-$CI_COMMIT_TAG.yaml
  tags:
    - prod
  only:
    - tags

修改完代码后可以在本地运行项目，看是否存在问题，这样可以在merge到master分支前避免低级错误。

# mvn clean package

# cd target/

# java -cp cronjob-demo-1.0-SNAPSHOT.jar com.mooc.demo.cronjob.Main

I will working for 29 seconds! It's very good!
All work is done! This is the first k8s ci/cd!

# cd ..

# rm -rf target/

可以看到，在本地运行该项目没有问题。

• push代码（harbor1）：

# git add .

# git commit -m 'Add .gitlab-ci.yml'

# git -c http.sslVerify=false push origin dev

直接发起合并申请，merge到master分支，

点击合并，CI/CD → 流水线，

测试环境项目部署完成，到master1节点查看，

# kubectl get pod |grep cronjob-demo
NAME                            READY   STATUS      RESTARTS   AGE
cronjob-demo-1574412360-9mmxx   0/1     Completed   0          2m34s
cronjob-demo-1574412420-qhlv5   0/1     Completed   0          94s
cronjob-demo-1574412480-6z587   0/1     Completed   0          34s

# kubectl logs cronjob-demo-1574412360-9mmxx 

I will working for 21 seconds! It's very good!
All work is done! This is the first k8s ci/cd!

# kubectl logs cronjob-demo-1574412420-qhlv5 

I will working for 24 seconds! It's very good!
All work is done! This is the first k8s ci/cd!

# kubectl logs cronjob-demo-1574412480-6z587 

I will working for 27 seconds! It's very good!
All work is done! This is the first k8s ci/cd!

可以看到，java的定时任务cronjob的输出与之前修改的代码一致，测试环境下部署没有问题。

CI/CD → 流水线，运行docker-build2及k8s-release任务，部署到预发布环境，

预发布环境项目部署完成，到master1节点查看，

# kubectl get pod |grep cronjob-release

cronjob-release-1574412600-kpjbv   0/1     Completed   0          2m23s
cronjob-release-1574412660-xk8rr   0/1     Completed   0          92s
cronjob-release-1574412720-zpvj5   0/1     Completed   0          32s

# kubectl logs cronjob-release-1574412600-kpjbv

I will working for 25 seconds! It's very good!
All work is done! This is the first k8s ci/cd!

# kubectl logs cronjob-release-1574412660-xk8rr

I will working for 18 seconds! It's very good!
All work is done! This is the first k8s ci/cd!

# kubectl logs cronjob-release-1574412720-zpvj5

I will working for 24 seconds! It's very good