Jenkins CD 流水线设计 Gitops CI制品信息保存为CD准备

富士康质检员张全蛋

已于 2024-03-02 09:57:10 修改

阅读量1.6k

点赞数 2

分类专栏： Devops CI/CD Jenkins 文章标签： jenkins

于 2021-09-24 15:23:54 首次发布

本文链接：https://blog.csdn.net/qq_34556414/article/details/120447240

版权

Devops CI/CD Jenkins 专栏收录该内容

81 篇文章 91 订阅

订阅专栏

下面这个阶段实在理解不了，可以将这个阶段去掉，如果公司不想这么用的话，大规模场景下使用也可以发现这种好处。

CI/CD流水线设计

总体目标：

我是一个用户，点开Jenkins之后输入版本分支，然后点流水线构建，第一个阶段下载代码，第二个阶段构建，生成了我们的包，第三个阶段进行代码扫描，第四个阶段是上传制品，此时这个制品到制品库里面去了，那么就到CD阶段了，也就是包在制品库了，可以进行去部署了。

传到制品库了，要将这个包下载下来，可以复制这个url的地址，现在就需要创建发布流水线，因为CI和CD都是不同的流水线，

这里包的版本号有几种格式

版本号1.1.1+commitid:拿到分支最后一次提交的ID，然后加上版本号，然后拼凑出最后的版本(这样是看不到历史版本的信息的)
CI上传制品，将制品的下载地址信息存起来，存储到git上面，在上传制品这种生成这个文件，这个文件里面存储了包的一些配置信息，CD流水线在拿的时候就拿这个信息（所以有两种方式，一种是存储信息，一种是不存储信息，不存储就拼接url直接下载下来就可以了，直接输入版本号就行）

提交了ReleaseFile之后想自动触发，存放到git上面去了，这次做了一个变更，git识别到了，这是一个push动作，那么会自动触发CD流水线去部署。如果做自动化可以这样去搞，不做自动化可以将这个步骤去掉。

我们将CI和CD分成两条流水线作业。

CI作业：用户输入版本分支后下载代码，进行构建扫描最终将制品上传到制品仓库, 生成版本文件。（在上传制品之后，还得加一个阶段，生成一个文件，这个ReleaseFile文件里面存储的就是包的一些配置信息，CD流水线拿到这个文件就可以了）
CD作业: 用户输入发布版本和选择要发布的主机IP后，下载制品，将制品和服务启动脚本cp到目标机器的发布目录，远程执行启动脚本启动服务并进行健康检查。

（1）CI 生成版本文件

在流水线最后一个步骤加上生成版本文件

获取gitlab上面的模板文件到本地（Gitlab下载文件的接口）
生成文件里面内容，将配置信息写入本地的一个文件当中（新版本文件，yaml文件的更改)
将这个文件传到gitlab（修改的是新版本文件）（gitlab上传文件的接口）

k8s里面的yaml文件都需要我们去更新，一般都是通过sed去修改yaml文件里面的内容，我们可以修改版本文件，这样就可以看到变更记录了。变更了啥信息都可以在git上面看得到。

最终效果如下：这个环境库里面存放着所有的发布的信息

发布信息的模板按照下面的去写

CD 获取版本文件（这样就可以看到变更的历史记录了）

1 从Gitlab下载版本文件

2 发布的时候只读取本地的版本文件

实践

先创建一个空项目，用来存储版本信息

新建一个模板文件叫release，根据实际业务想存储什么信息可以自己去定义。

buname:  业务名称
appname: 应用名称
version: 制品版本
artifact: 制品下载链接

现在要修改这个文件的内容，就是要先下载下来，然后上传。

现在创建token

这个token拿到了

现在找到gitlab的api

我这里的实现是用HttpRequest插件来实现的，没有使用curl

[root@localhost ~]# curl --header "PRIVATE-TOKEN:SCPnzayH7YivMxaR1Yy-" "http://192.168.11.129/api/v4/projects/9/repository/files/release.yaml/raw?ref=master" -o test.yml
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    64  100    64    0     0   1369      0 --:--:-- --:--:-- --:--:--  1391

[root@localhost ~]# curl --header "PRIVATE-TOKEN:SCPnzayH7YivMxaR1Yy-" "http://192.168.11.129/api/v4/projects/9/repository/files/release.yaml/raw?ref=master"
buname: _NULL_
appname: _NULL_
version: _NULL_
artifact: _NULL_

curl --location --request GET 'http://192.168.11.129/api/v4/projects/9/repository/files/release.yaml/raw?ref=master' \
--header 'Authorization: Bearer SCPnzayH7YivMxaR1Yy-'

Repository files API | GitLabhttps://docs.gitlab.com/ee/api/repository_files.html

现在要去下载这个文件，需要project ID和分支名称和文件名称。

pipeline {
    agent any

    stages {
        stage('Hello') {
            steps {
                    script {
                    //获取版本库文件内容 devops02-env/release.yaml
                    response = GetRepoFile(17,"release.yaml", "main")
                    println(response)
                    
                }
            }
        }
    }
}


// 封装HTTP
def HttpReq(reqType, reqUrl,reqBody ){
            def gitServer = "http://139.198.166.235:81/api/v4"
            withCredentials([string(credentialsId: 'ecbcd399-da69-4802-8760-87a1c1ff58a1', variable: 'GITLABTOKEN')]) {
                response = httpRequest acceptType: 'APPLICATION_JSON_UTF8', 
                                consoleLogResponseBody: true, 
                                contentType: 'APPLICATION_JSON_UTF8', 
                                customHeaders: [[maskValue: false, name: 'PRIVATE-TOKEN', value: "${GITLABTOKEN}"]], 
                                httpMode: "${reqType}", 
                                url: "${gitServer}/${reqUrl}", 
                                wrapAsMultipart: false,
                                requestBody: "${reqBody}"

    }
    return response
}


//获取文件内容
def GetRepoFile(projectId,filePath, branchName ){
   //GET /projects/:id/repository/files/:file_path/raw
   apiUrl = "/projects/${projectId}/repository/files/${filePath}/raw?ref=${branchName}"
   response = HttpReq('GET', apiUrl, "")

   return response.content

}

结果如下：

Started by user admin
Running in Durability level: MAX_SURVIVABILITY
[Pipeline] Start of Pipeline
[Pipeline] node
Running on build-01 in /data/cicd/jenkinsagent/workspace/nexus/release-file
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Hello)
[Pipeline] script
[Pipeline] {
[Pipeline] withCredentials
Masking supported pattern matches of $GITLABTOKEN
[Pipeline] {
[Pipeline] httpRequest
Warning: A secret was passed to "httpRequest" using Groovy String interpolation, which is insecure.
		 Affected argument(s) used the following variable(s): [GITLABTOKEN]
		 See https://jenkins.io/redirect/groovy-string-interpolation for details.
HttpMethod: GET
URL: http://139.198.166.235:81/api/v4//projects/17/repository/files/release.yaml/raw?ref=main
Content-Type: application/json; charset=UTF-8
Accept: application/json
PRIVATE-TOKEN: ****
Sending request to url: http://139.198.166.235:81/api/v4//projects/17/repository/files/release.yaml/raw?ref=main
Response Code: HTTP/1.1 200 OK
Response: 
buname: _NULL_
appname: _NULL_
version: _NULL_
artifact: _NULL_

Success: Status code 200 is in the accepted range: 100:399
[Pipeline] }
[Pipeline] // withCredentials
[Pipeline] echo
buname: _NULL_
appname: _NULL_
version: _NULL_
artifact: _NULL_

[Pipeline] }
[Pipeline] // script
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
Finished: SUCCESS

因为获取文件内容返回的是yaml格式的内容，以前返回的是json，现在用的yaml

//这里的分支版本可以通过branchName再做一个split就可以拿到，比如branchName:relaese-1.1.1
//可以在上传制品的时候返回artifactUrl信息
env.buName = "acmp"
env.appName = "myapp"
env.releaseVersion = "1.1.1"
env.artifactUrl = "http://139.198.166.235:8082/repository/devops-repo/acmp/acmp-myapp-service/1.1.1/acmp-myapp-service-1.1.1.jar"



pipeline {
    agent any


    stages {
        stage('Hello') {
            steps {
                    script {
                    //获取版本库文件内容 devops02-env/release.yaml
                    response = GetRepoFile(17,"release.yaml", "main")
                    //println(response)
                    yamlData = readYaml text: """${response}"""
                    yamlData.version   = "${env.releaseVersion}"
                    yamlData.artifact  = "${env.artifactUrl}"
                    yamlData.buname    = "${env.buName}"
                    yamlData.appname   = "${env.appName}"
                    println(yamlData.toString())
                    
                }
            }
        }
    }
}


def HttpReq(reqType, reqUrl,reqBody ){
            def gitServer = "http://139.198.166.235:81/api/v4"
            withCredentials([string(credentialsId: 'ecbcd399-da69-4802-8760-87a1c1ff58a1', variable: 'GITLABTOKEN')]) {
                response = httpRequest acceptType: 'APPLICATION_JSON_UTF8', 
                                consoleLogResponseBody: true, 
                                contentType: 'APPLICATION_JSON_UTF8', 
                                customHeaders: [[maskValue: false, name: 'PRIVATE-TOKEN', value: "${GITLABTOKEN}"]], 
                                httpMode: "${reqType}", 
                                url: "${gitServer}/${reqUrl}", 
                                wrapAsMultipart: false,
                                requestBody: "${reqBody}"

    }
    return response
}


//获取文件内容
def GetRepoFile(projectId,filePath, branchName ){
   //GET /projects/:id/repository/files/:file_path/raw
   apiUrl = "/projects/${projectId}/repository/files/${filePath}/raw?ref=${branchName}"
   response = HttpReq('GET', apiUrl, "")

   return response.content

}

Started by user admin
Running in Durability level: MAX_SURVIVABILITY
[Pipeline] Start of Pipeline
[Pipeline] node
Running on build-01 in /data/cicd/jenkinsagent/workspace/nexus/release-file
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Hello)
[Pipeline] script
[Pipeline] {
[Pipeline] withCredentials
Masking supported pattern matches of $GITLABTOKEN
[Pipeline] {
[Pipeline] httpRequest
Warning: A secret was passed to "httpRequest" using Groovy String interpolation, which is insecure.
		 Affected argument(s) used the following variable(s): [GITLABTOKEN]
		 See https://jenkins.io/redirect/groovy-string-interpolation for details.
HttpMethod: GET
URL: http://139.198.166.235:81/api/v4//projects/17/repository/files/release.yaml/raw?ref=main
Content-Type: application/json; charset=UTF-8
Accept: application/json
PRIVATE-TOKEN: ****
Sending request to url: http://139.198.166.235:81/api/v4//projects/17/repository/files/release.yaml/raw?ref=main
Response Code: HTTP/1.1 200 OK
Response: 
buname: _NULL_
appname: _NULL_
version: _NULL_
artifact: _NULL_

Success: Status code 200 is in the accepted range: 100:399
[Pipeline] }
[Pipeline] // withCredentials
[Pipeline] readYaml
[Pipeline] echo
[buname:acmp, appname:myapp, version:1.1.1, artifact:http://139.198.166.235:8082/repository/devops-repo/acmp/acmp-myapp-service/1.1.1/acmp-myapp-service-1.1.1.jar]
[Pipeline] }
[Pipeline] // script
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
Finished: SUCCESS

现在就可以去替换文件里面内容，然后上传

env.buName = "acmp"
env.appName = "myapp"
env.releaseVersion = "1.1.1"
env.artifactUrl = "http://139.198.166.235:8082/repository/devops-repo/acmp/acmp-myapp-service/1.1.1/acmp-myapp-service-1.1.1.jar"
env.branchName = "release-1.1.1"


pipeline {
    agent any


    stages {
        stage('Hello') {
            steps {
                    script {
                    //下载版本库文件 devops02-env/release.yaml
                    response = GetRepoFile(17,"release.yaml", "main")
                    //println(response)
                    yamlData = readYaml text: """${response}"""
                    yamlData.version   = "${env.releaseVersion}"
                    yamlData.artifact  = "${env.artifactUrl}"
                    yamlData.buname    = "${env.buName}"
                    yamlData.appname   = "${env.appName}"
                    println(yamlData.toString())

                    sh "ls && rm -fr test.yaml"
                    writeYaml charset: 'UTF-8', data: yamlData, file: 'test.yaml'
                    newYaml = sh returnStdout: true, script: 'cat test.yaml'
                    
                    println(newYaml)
                    //更新gitlab文件内容,转化为base64，在调用api上传的时候都是base64编码
                    base64Content = newYaml.bytes.encodeBase64().toString()

                    // 会有并行问题，同时更新报错
                    try {
                        UpdateRepoFile(17,"${env.appName}%2f${env.branchName}.yaml",base64Content, "main")
                    } catch(e){
                        CreateRepoFile(17,"${env.appName}%2f${env.branchName}.yaml",base64Content, "main")
                    }
                    
                }
            }
        }
    }
}


def HttpReq(reqType, reqUrl,reqBody ){
            def gitServer = "http://139.198.166.235:81/api/v4"
            withCredentials([string(credentialsId: 'ecbcd399-da69-4802-8760-87a1c1ff58a1', variable: 'GITLABTOKEN')]) {
                response = httpRequest acceptType: 'APPLICATION_JSON_UTF8', 
                                consoleLogResponseBody: true, 
                                contentType: 'APPLICATION_JSON_UTF8', 
                                customHeaders: [[maskValue: false, name: 'PRIVATE-TOKEN', value: "${GITLABTOKEN}"]], 
                                httpMode: "${reqType}", 
                                url: "${gitServer}/${reqUrl}", 
                                wrapAsMultipart: false,
                                requestBody: "${reqBody}"

    }
    return response
}


//获取文件内容
def GetRepoFile(projectId,filePath, branchName ){
   //GET /projects/:id/repository/files/:file_path/raw
   apiUrl = "/projects/${projectId}/repository/files/${filePath}/raw?ref=${branchName}"
   response = HttpReq('GET', apiUrl, "")

   return response.content

}


//更新文件内容
def UpdateRepoFile(projectId,filePath,fileContent, branchName){
    apiUrl = "projects/${projectId}/repository/files/${filePath}"
    reqBody = """{"branch": "${branchName}","encoding":"base64", "content": "${fileContent}", "commit_message": "update a new file"}"""
    response = HttpReq('PUT',apiUrl,reqBody)
    println(response)

}

//创建文件
def CreateRepoFile(projectId,filePath,fileContent, branchName){
    apiUrl = "projects/${projectId}/repository/files/${filePath}"
    reqBody = """{"branch": "${branchName}","encoding":"base64", "content": "${fileContent}", "commit_message": "update a new file"}"""
    response = HttpReq('POST',apiUrl,reqBody)
    println(response)

}

[root@localhost ~]# curl --request POST --header 'PRIVATE-TOKEN: SCPnzayH7YivMxaR1Yy-'      --headebranch": "master", "author_email": "author@example.com", "author_name": "Firstname Lastname",
               "content": "buname: _NULL_ \nappname: _NULL_\nversion: _NULL_\nartifact: _NULL_", "commit_message": "create a new file"}'      "http://192.168.11.129/api/v4/projects/9/repository/files/release-1.6.yaml"
{"file_path":"release-1.6.yaml","branch":"master"}[root@localhost ~]# 



curl --request POST --header 'PRIVATE-TOKEN: <your_access_token>' \
     --header "Content-Type: application/json" \
     --data '{"branch": "master", "author_email": "author@example.com", "author_name": "Firstname Lastname",
               "content": "some content", "commit_message": "create a new file"}' \
     "https://gitlab.example.com/api/v4/projects/13083/repository/files/app%2Fproject%2Erb"

${env.appName}%2f${env.branchName}.yaml

上面意思是放在17号仓库，仓库下面文件夹名字为${env.appName}，在该文件夹下面有生成一个文件{env.branchName}.yaml。这里使用的是%2f，本来是目录/文件这种格式，但是编码这里转换为了%2f，所以转化为编码要不然会失败。

(17,"${env.appName}%2f${env.branchName}.yaml",base64Content, "main")

仓库id+文件夹+文件+文件里面内容+分支名称

最后结果如下，可以看到符合预期，在版本信息管理库devops-env下面生成了对应项目的目录，并且目录下面包含制品的信息：

到时候CD的时候就可以拿下该文件进行发布了，拿下该文件，通过脚本对这个文件处理一下，拿到制品下载地址就可以下载下来了。

上面就是调用gitlab api去实现这个过程。

Gitops其实就是为环境单独创建了一个git仓库，无非就是在原有的CI的基础上面，加了一个步骤生成了一个文件，把项目的以及代码的信息全部都放在这个文件里面了（后期拿到这个文件能够获取里面的参数去部署），然后在这里面做了一个变更，此时自动触发到我们的环境里面，好处是每次的变更都可以看到。

在k8s里面，全部都是yaml文件，我们一般改的就是镜像，如果是helm的话，修改的是values.yaml文件，总之就是更新里面的内容，更新了之后去发布。

如果接受不了上面的步骤，那么就不要版本文件了，直接拿制品自己拼接就行了。

总结：所谓的gitops就是把部署描述文件，存放到git系统里面，ci的时候自动更新，cd可以通过仓库自动触发。

完整的代码如下

package org.devops


// 封装HTTP
def HttpReq(reqType, reqUrl,reqBody ){
    def gitServer = "http://139.198.166.235:81/api/v4"
    withCredentials([string(credentialsId: 'ecbcd399-da69-4802-8760-87a1c1ff58a1', variable: 'GITLABTOKEN')]) {
        response = httpRequest acceptType: 'APPLICATION_JSON_UTF8', 
                          consoleLogResponseBody: true, 
                          contentType: 'APPLICATION_JSON_UTF8', 
                          customHeaders: [[maskValue: false, name: 'PRIVATE-TOKEN', value: "${GITLABTOKEN}"]], 
                          httpMode: "${reqType}", 
                          url: "${gitServer}/${reqUrl}", 
                          wrapAsMultipart: false,
                          requestBody: "${reqBody}"

    }
    return response
}

//获取文件内容
def GetRepoFile(projectId,filePath, branchName ){
   //GET /projects/:id/repository/files/:file_path/raw
   apiUrl = "/projects/${projectId}/repository/files/${filePath}/raw?ref=${branchName}"
   response = HttpReq('GET', apiUrl, "")

   return response.content

}

//更新文件内容
def UpdateRepoFile(projectId,filePath,fileContent, branchName){
    apiUrl = "projects/${projectId}/repository/files/${filePath}"
    reqBody = """{"branch": "${branchName}","encoding":"base64", "content": "${fileContent}", "commit_message": "update a new file"}"""
    response = HttpReq('PUT',apiUrl,reqBody)
    println(response)

}

//创建文件
def CreateRepoFile(projectId,filePath,fileContent, branchName){
    apiUrl = "projects/${projectId}/repository/files/${filePath}"
    reqBody = """{"branch": "${branchName}","encoding":"base64", "content": "${fileContent}", "commit_message": "update a new file"}"""
    response = HttpReq('POST',apiUrl,reqBody)
    println(response)

}

script {
                    //下载版本库文件 devops02-env/release.yaml
                    response = GetRepoFile(17,"release.yaml", "main")
                    //println(response)
                    yamlData = readYaml text: """${response}"""
                    yamlData.version   = "${env.releaseVersion}"
                    yamlData.artifact  = "${env.artifactUrl}"
                    yamlData.buname    = "${env.buName}"
                    yamlData.appname   = "${env.appName}"
                    println(yamlData.toString())

                    sh "ls && rm -fr test.yaml"
                    writeYaml charset: 'UTF-8', data: yamlData, file: 'test.yaml'
                    newYaml = sh returnStdout: true, script: 'cat test.yaml'
                    
                    println(newYaml)
                    //更新gitlab文件内容,转化为base64，在调用api上传的时候都是base64编码
                    base64Content = newYaml.bytes.encodeBase64().toString()

                    // 会有并行问题，同时更新报错
                    try {
                        UpdateRepoFile(17,"${env.appName}%2f${env.branchName}.yaml",base64Content, "main")
                    } catch(e){
                        CreateRepoFile(17,"${env.appName}%2f${env.branchName}.yaml",base64Content, "main")
                    }
                    
                }