实践:devops之云主机模式持续部署(ci-cd)

实践:devops之云主机模式持续部署(ci-cd)

tstmp_20230701223729

目录

image-20230702201542710

推荐文章

https://www.yuque.com/xyy-onlyone/aevhhf?# 《玩转Typora》

image-20230624094222589

0、流程分析

2条Jenkins pipeline

tstmp_20230701223729

image-20230702074550297

CI pipeline

image-20230702194213612

image-20230702200010478

CD pipeline

image-20230702194227921

image-20230702200026808

标准规范

image-20230702074252134

项目规范与总体设计

公司里面要使用流水线要做持续集成CI/CD的项目越来越多,这对流水线的设计和开发有不同的要求。我们经常听到用户的反馈:

  • 各种不同语言的技术栈, 如何使流水线适配呢? 从不同技术栈维护一套流水线模版,到我们使用共享库进行统一的管理和维护。

  • 对于不同的项目,大家管理代码的方式也不同。可能还有一部分用户在使用Svn等不同的版本控制系统。

  • 不同的项目,开发模式也不太一样, 编译构建工具不同,发布的方式也有不同的地方…

等等,不止上面的问题。所以在做流水线的使用应该提前把项目团队的规范定义好, 这样后期项目改造后可以直接集成CI/CD流水线。更加便捷。

1.团队信息

信息项描述
业务简称/编号devops4
开发模式特性分支开发,版本分支发布,主干分支作为最新代码
项目类型与构建方式前端: vue项目, npm打包, 制品目录 dist
后端:springboot项目, maven打包, 制品目录 target
发布主机环境(vm)LB: 192.168.1.200
Server: 192.168.1.121~192.168.1.122

2.CI/CD规范

通过上面的信息,我们采用如下规范:

工具链
GitLab 代码库仓库组: devops4
项目仓库后端 devops4-ops-service 前端 devops4-ops-ui
Jenkins作业文件夹: devops4
作业命名: 后端 devops4-ops-service 前端 devops4-ops-ui
CI构建规范前端项目采用npm打包后统一放到dist目录下, 静态文件以tgz打包。
后端项目采用maven打包后统一放到target目录下,以jar包。
Sonar代码报告前端项目:devops4-ops-ui 后端项目:devops4-ops-service
项目团队可以使用devops4命名的自定义质量规则和质量阈。
Nexus制品库目录
devops4-ops-service/version/devops4-ops-service-version.jar
devops4-ops-ui/version/devops4-ops-ui-version.tar.gz
devops4版本: 分割release分支获取版本号
发布规范用户输入版本,下载制品库,使用脚本启动服务。

标准化

版本分支命名:RELEASE-1.1.1

分支策略

特性分支开发,版本分支发布。

tstmp_20230702064028

环境管理

使用virtualbox创建2台虚拟机, 或者采用terraform操作云平台创建2台虚机。

本次,自己使用2台本地vmwareworkstation虚机测试。

tstmp_20230702064109

制品管理

制品版本命名:版本号-CommitID

tstmp_20230702064208

发布流水线

Jenkins pipeline * 2

  • CI pipeline

  • CD pipeline

    • 复选框参数: 发布主机
    • 字符参数:版本分支
    • 选项参数:目标环境 【dev/uat/stg/prod】

tstmp_20230702064303

应用发布与回滚策略

Deploy发布策略

image-20230702074339936

蓝绿发布

环境存在两个版本,蓝版本和绿版本同时存在,部署新版本然后进行测试,将流量切到新版本,最终实际运行的只有一个版本(蓝/绿)。好处是无需停机,并且发布风险较小。

img

nginx upstream模块实现:

upstream webservers {
    server 192.168.1.253:8099 weight=100;
    server 192.168.1.252:8099 down;
}

server {
      listen       8017; 
      location / {
        proxy_pass http://webservers;
      }
}

nginx -s reload
灰度发布

将发行版发布到一部分用户或服务器的一种模式。这个想法是首先将更改部署到一小部分服务器,进行测试,然后将更改推广到其余服务器。一旦通过所有运行状况检查,当没有问题时,所有的客户将被路由到该应用程序的新版本,而旧版本将被删除。

img

img

nginx 权重模拟:

upstream webservers {
    server 192.168.1.223:8099 weight=100;
    server 192.168.1.222:8099 weight=100;
    server 192.168.1.221:8099 weight=100;
}

server {
      listen       8017; 
      location / {
        proxy_pass http://webservers;
      }
}

nginx -s reload
版本回滚
  • 版本一直升级,则无需回滚。
  • 选择旧版本文件,进行发布。

前端后端项目发布

1、前端项目

复制静态文件到nginx站点目录,nginx -s reload

## 进入Web服务器的站点目录下

## 下载包
[root@master html]# curl -u admin:admin123 http://192.168.1.200:8081/repository/anyops/com/anyops/anyops-devops-ui/1.1.1/anyops-devops-ui-1.1.1.tar.gz  -o anyops-devops-ui-1.1.1.tar.gz 
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  196k  100  196k    0     0  24.0M      0 --:--:-- --:--:-- --:--:-- 24.0M

## 解压包
[root@master html]# tar zxf anyops-devops-ui-1.1.1.tar.gz 
[root@master html]# ls
anyops-devops-ui-1.1.1.tar.gz  index.html  static

## 触发nginx重载
[root@master html]# nginx -s reload

tstmp_20230702072623

2、后端项目

  • 复制jar包到目标目录, 使用nohup java -jar 启动服务。
  • nohup java -jar app.jar >output 2>&1 &

tstmp_20230702072810

1、CI

拷贝Jenkins流水线

  • 拷贝Jenkins作业devops6-maven-servicedevops6-maven-service_CI

image-20230630124853726

保存后,点击参数化构建,会发现branchName的页面参数为空,我们先直接运行一次流水线看看效果:

运行一次流水线后,再次运行时,就会看到branchName正常了。

接下来我们就用devops6-maven-service_CI来测试。

image-20230630125131260

  • 我们再次运行下,看下效果

image-20230630125359978

可以看到,流水线运行成功。

image-20230630125435262

image-20230630125458519

可以看到nexus仓库里制品被上传成功了。

优化pipeline代码,去除制品库里CI字样

image-20230630125913459

                    appName = "${JOB_NAME}".split('_')[0] //devops6-maven-service_CI
                    repoName = appName.split('-')[0]   //devops6
                    appVersion = "${env.branchName}".split("-")[-1] // RELEASE-1.1.1   1.1.1
                    targetDir="${appName}/${appVersion}"

再次运行测试效果:

image-20230630130045837

image-20230630130137478

符合预期。

新建Jenkins CD流水线

  • 创建一个devops6-maven-service_CD作业,然后添加一些页面参数

image-20230630212933661

image-20230630212942420

image-20230630212950308

image-20230630212959078

点击参数化构建:

image-20230630212847814

创建一个devops6的视图

image-20230630214923752

image-20230630214939599

image-20230630214955849

优化pipeline代码,nexus仓库的版本里要带上commitID

  • 之前仓库是这样的

image-20230630213211572

  • 先来手动获取下项目仓库的commitID

image-20230701062133819

[root@Devops6 ~]#cd /opt/jenkinsagent/workspace/
[root@Devops6 workspace]#ls
day2-pipeline-demo      devops6-gradle-service      devops6-maven-service     devops6-maven-service_CI@tmp  devops6-maven-test      devops6-npm-service      test-maven    
day2-pipeline-demo@tmp  devops6-gradle-service@tmp  devops6-maven-service_CI  devops6-maven-service@tmp     devops6-maven-test@tmp  devops6-npm-service@tmp  test-maven@tmp
[root@Devops6 workspace]#cd devops6-maven-service_CI
[root@Devops6 devops6-maven-service_CI]#ls
mvnw  mvnw.cmd  pom.xml  sonar-project.properties  src  target
[root@Devops6 devops6-maven-service_CI]#git rev-parse HEAD #通过这个命令之可以获取仓库comitID的。
b5cfb8eeee597edd752cb11f5daa9ac843fb9f97
[root@Devops6 devops6-maven-service_CI]#

然后利用片段生成器生成代码:

image-20230701062623617

sh returnStdout: true, script: 'git rev-parse HEAD'

然后集成到piepeline代码里。

  • 我们想让这里的版本号也带上commitID

这里直接写代码:

appVersion = "${appVersion}-${env.commitID}"

//获取commitID
env.commitID = gitlab.GetCommitID()
println("commitID: ${env.commitID}")


package org.devops

//获取CommitID
def GetCommitID(){
    ID = sh returnStdout: true, script:"git rev-parse HEAD"
    ID = ID -"\n"
    return ID[0..7] //取前8位id
}

image-20230630214612592

image-20230630214641609

image-20230630214733321

  • 在gitlab的devops6-maven-service里以main分支创建RELEASE-9.9.9分支

image-20230630214020268

  • 运行devops6-maven-service_CI流水线

image-20230630214129250

image-20230630214320025

image-20230630214335518

image-20230630214418763

image-20230630214452618

测试成功。

2、CD

下载制品

cd部分就不用再下载代码获取commitID了。

我们来使用gitlab api获取分支commit。

Step1: 获取GitLab 分支CommitID

  • 打开gitlab api官方文档

https://docs.gitlab.com/

image-20230701063900336

image-20230701064429462

curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/5/repository/branches/main"
  • 在postman里调试

先拿取下一些参数:

这里拿到Project ID

image-20230701064854914

然后在gitlab上创建一个token:

image-20230701065021346

image-20230701065059533

TLDgj3sz-cioyk6AfxZi

调试:

http://172.29.9.101:8076/api/v4/projects/7/repository/branches/RELEASE-9.9.9

添加get请求,添加PRIVARE-TOKEN,点击Send。

image-20230701065406859

此时,我们就通过gitalb api拿到了分支commitID了,和之前手动执行git命令获取的commitID信息一致。

image-20230701065603410

image-20230701065732647

  • 此时拿到postman给出的cURL命令

image-20230701065902230

curl --location 'http://172.29.9.101:8076/api/v4/projects/7/repository/branches/RELEASE-9.9.9' \
--header 'PRIVATE-TOKEN: TLDgj3sz-cioyk6AfxZi'
  • 优化pipeline代码

创建Gitlab.groovy文件

package org.devops

//发起HTTP请求
def HttpReq(method, apiUrl){
    response = sh  returnStdout: true, 
        script: """ 
        curl --location --request ${method} \
        http:172.29.9.101:8076/api/v4/${apiUrl} \
        --header "PRIVATE-TOKEN: TLDgj3sz-cioyk6AfxZi"
    """
    response = readJSON text: response - "\n" //json数据的读取方式
    return response
}

image-20230701070729445

但是,这里的gitlab token是明文的,因此需要在jenkins里配置个凭据。

image-20230701070836183

image-20230701070942488

image-20230701070956541

然后利用片段生成器来利用次token,生成代码:

image-20230701071159724

withCredentials([string(credentialsId: '5782c77d-ce9d-44e5-b9ba-1ba2097fc31d', variable: 'gitlabtoken')]) {
    // some block
}
  • 优化pipeline代码
package org.devops

//发起HTTP请求
def HttpReq(method, apiUrl){

    withCredentials([string(credentialsId: '5782c77d-ce9d-44e5-b9ba-1ba2097fc31d', variable: 'gitlabtoken')]) {
        response = sh  returnStdout: true, 
        script: """ 
        curl --location --request ${method} \
        http:172.29.9.101:8076/api/v4/${apiUrl} \
        --header "PRIVATE-TOKEN: ${gitlabtoken}"
        """
    }
    response = readJSON text: response - "\n" //json数据的读取方式
    return response
}

image-20230701071409361

但是,存在一个问题,apiUrl里我们还需要知道ProjectID才行,这里继续查找gitlab api。

  • 获取ProjectID

image-20230701073750499

image-20230701074248170

http://172.29.9.101:8076/api/v4/projects?search=devops6-maven-service

curl --location 'http://172.29.9.101:8076/api/v4/projects?search=devops6-maven-service' \
--header 'PRIVATE-TOKEN: TLDgj3sz-cioyk6AfxZi'

image-20230701074437455

  • 继续优化pipeline代码
package org.devops

//发起HTTP请求
def HttpReq(method, apiUrl){

    withCredentials([string(credentialsId: '5782c77d-ce9d-44e5-b9ba-1ba2097fc31d', variable: 'gitlabtoken')]) {
        response = sh  returnStdout: true, 
        script: """ 
        curl --location --request ${method} \
        http:172.29.9.101:8076/api/v4/${apiUrl} \
        --header "PRIVATE-TOKEN: ${gitlabtoken}"
        """
    }
    response = readJSON text: response - "\n" //json数据的读取方式
    return response
}

//获取ProjectID
def GetProjectIDByName(projectName, groupName){
    apiUrl = "projects?search=${projectName}"
    response = HttpReq("GET", apiUrl)
    if (response != []){
        for (p in response) {
            if (p["namespace"]["name"] == groupName){
                return response[0]["id"]
            }
        }
    }
}

//获取分支CommitID
def GetBranchCommitID(projectID, branchName){
    apiUrl = "projects/${projectID}/repository/branches/${branchName}"
    response = HttpReq("GET", apiUrl)
    return response.commit.short_id
}
  • 创建cd.jenkinsfile

image-20230701082636305

@Library("devops06@main") _ 

//import  src/org/devops/Gitlab.groovy
def mygit = new org.devops.Gitlab()


//pipeline
pipeline{
    agent { label "build"}
    options {
        skipDefaultCheckout true
    }
    stages{
        stage("GetArtifact"){
            steps{
                script{
                    env.projectName = "${JOB_NAME}".split('_')[0] //devops6-maven-service
                    env.groupName = "${env.projectName}".split('-')[0]  //devops6
                    
                    projectID = mygit.GetProjectIDByName(env.projectName, env.groupName)
                    commitID = mygit.GetBranchCommitID("${projectID}", "${env.branchName}")
                    println(commitID)

                    // appVersion = "${env.branchName}".split("-")[-1]  //9.9.9
                    // println(appVersion)
                    // currentBuild.description = "Version: ${appVersion}-${commitID}"

                    currentBuild.displayName = "第${BUILD_NUMBER}次构建-${commitID}"
                    currentBuild.description = "构建分支名称:${env.branchName}"
                
                }
            }
        }

    }
}

Gitlab.groovy代码

package org.devops

//发起HTTP请求
def HttpReq(method, apiUrl){

    withCredentials([string(credentialsId: '5782c77d-ce9d-44e5-b9ba-1ba2097fc31d', variable: 'gitlabtoken')]) {
        response = sh  returnStdout: true, 
        script: """ 
        curl --location --request ${method} \
        http://172.29.9.101:8076/api/v4/${apiUrl} \
        --header "PRIVATE-TOKEN: ${gitlabtoken}"
        """
    }
    response = readJSON text: response - "\n" //json数据的读取方式
    return response
}

//获取ProjectID
def GetProjectIDByName(projectName, groupName){
    apiUrl = "projects?search=${projectName}"
    response = HttpReq("GET", apiUrl)
    if (response != []){
        for (p in response) {
            if (p["namespace"]["name"] == groupName){
                return response[0]["id"]
            }
        }
    }
}

//获取分支CommitID
def GetBranchCommitID(projectID, branchName){
    apiUrl = "projects/${projectID}/repository/branches/${branchName}"
    response = HttpReq("GET", apiUrl)
    return response.commit.short_id
}
  • 编辑devops6-maven-service_CD流水线使用共享库

image-20230701082755595

image-20230701082806637

运行流水线:

image-20230701082824215

image-20230701082839197

测试成功。😘

Step2: 下载制品

  • nexus仓库制品地址如下
http://172.29.9.101:8081/repository/devops6/devops6-maven-service/6.1.1/devops6-maven-service-6.1.1.jar
  • 这里编写pipeline代码
@Library("devops06@main") _ 

//import  src/org/devops/Gitlab.groovy
def mygit = new org.devops.Gitlab()


//pipeline
pipeline{
    agent { label "build"}
    options {
        skipDefaultCheckout true
    }
    stages{
        stage("GetArtifact"){
            steps{
                script{
                    env.projectName = "${JOB_NAME}".split('_')[0] //devops6-maven-service
                    env.groupName = "${env.projectName}".split('-')[0]  //devops6
                    
                    projectID = mygit.GetProjectIDByName(env.projectName, env.groupName)
                    commitID = mygit.GetBranchCommitID("${projectID}", "${env.branchName}")
                    println(commitID)

                    appVersion = "${env.branchName}".split("-")[-1]  //9.9.9
                    println(appVersion)
                    // currentBuild.description = "Version: ${appVersion}-${commitID}"

                    currentBuild.displayName = "第${BUILD_NUMBER}次构建-${commitID}"
                    currentBuild.description = "构建分支名称:${env.branchName}"


                    //下载制品
                    //http://172.29.9.101:8081/repository/devops6/devops6-maven-service/6.1.1-b5cfb8ee/devops6-maven-service-6.1.1-b5cfb8ee.jar
                    repoUrl = "http://172.29.9.101:8081/repository/${env.groupName}"
                    artifactName = "${env.projectName}-${appVersion}-${commitID}.jar"
                    artifactUrl = "${repoUrl}/${env.projectName}/${appVersion}-${commitID}/${artifactName}"
                    sh "wget --no-verbose ${artifactUrl} && ls -l"
                
                }
            }
        }

    }
}

image-20230701090006744

  • 运行观察效果

image-20230701085928107

image-20230701085915156

下载制品成功。

我们再运行一次流水线:

image-20230701090110353

会看到多了一个包,

最后我们发布完,会把它清掉的:

这里先手动给清掉。

image-20230701090216170

Step3: 发布

准备2台linux机器
devops-deploy1-172.29.9.110
devops-deploy2-172.29.9.111
  • 给这2台机器装好java-11
yum install -y java-11-openjdk.x86_64
devops机器安装ansible环境
yum install epel-release  -y
yum install ansible -y
  • 编辑下ansible的主机管理文件:
[root@Devops6 ~]#vim /etc/ansible/hosts
172.29.9.110
172.29.9.111
  • 给ansible机器到2个节点做个免密
ssh-keygen
ssh-copy-id -i ~/.ssh/id_rsa.pub root@172.29.9.110
ssh-copy-id -i ~/.ssh/id_rsa.pub root@172.29.9.111
  • 查看当前主机是否在线:
[root@Devops6 ~]#ansible all  -m ping -u root

image-20230701191542756

构建一次devops6-maven-service_CD,下载制品

image-20230701191814077

image-20230701191928480

我们先来手动发布一次,再集成到CD流水线里
  • 拷贝制品到deploy1
[root@Devops6 devops6-maven-service_CD]#ansible 172.29.9.110 -m copy -a "src=devops6-maven-service-9.9.9-b5cfb8ee.jar  dest=/opt/devops6-maven-service-9.9.9-b5cfb8ee.jar"

image-20230701201005371

image-20230701201043602

  • 启动服务:

image-20230701201734093

image-20230701201803075

  • 用准备好的服务启动脚本来启动/停止java服务

服务启动脚本:service.sh (原始脚本如下)

#!/bin/bash

# sh service.sh anyops-devops-service 1.1.1 8091 start
APPNAME=NULL
VERSION=NULL
PORT=NULL

start(){
    port_result=`netstat -anlpt | grep "${PORT}" || echo false`

    if [[ $port_result == "false" ]];then
        nohup java -jar -Dserver.port=${PORT}  ${APPNAME}-${VERSION}.jar >${APPNAME}.log.txt 2>&1 &
    else
       stop
       sleep 5
       nohup java -jar -Dserver.port=${PORT}  ${APPNAME}-${VERSION}.jar >${APPNAME}.log.txt 2>&1 &
    fi
}


stop(){
    pid=`netstat -anlpt | grep "${PORT}" | awk '{print $NF}' | awk -F '/' '{print $1}' | head -1`
    kill -15 $pid
}


check(){
    proc_result=`ps aux | grep java | grep "${APPNAME}" | grep -v grep || echo false`
    port_result=`netstat -anlpt | grep "${PORT}" || echo false`
    url_result=`curl -s http://localhost:${PORT} || echo false `

    if [[ $proc_result == "false" || $port_result == "false" || $url_result == "false" ]];then
        echo "server not running"
    else
        echo "ok"
    fi
}

case $1 in
    start)
        start
        sleep 5
        check
        ;;

    stop)
        stop
        sleep 5
        check
        ;;
    restart)
        stop
        sleep 5
        start
        sleep 5
        check
        ;;
    check)
        check
        ;;
    *)
        echo "sh service.sh {start|stop|restart|check}"
        ;;
esac

参数写入后脚本如下

#!/bin/bash

# sh service.sh anyops-devops-service 1.1.1 8091 start
APPNAME=devops6-maven-service
VERSION=9.9.9-b5cfb8ee
PORT=8080

start(){
    port_result=`netstat -anlpt | grep "${PORT}" || echo false`

    if [[ $port_result == "false" ]];then
        nohup java -jar -Dserver.port=${PORT}  ${APPNAME}-${VERSION}.jar >${APPNAME}.log.txt 2>&1 &
    else
       stop
       sleep 5
       nohup java -jar -Dserver.port=${PORT}  ${APPNAME}-${VERSION}.jar >${APPNAME}.log.txt 2>&1 &
    fi
}


stop(){
    pid=`netstat -anlpt | grep "${PORT}" | awk '{print $NF}' | awk -F '/' '{print $1}' | head -1`
    kill -15 $pid
}


check(){
    proc_result=`ps aux | grep java | grep "${APPNAME}" | grep -v grep || echo false`
    port_result=`netstat -anlpt | grep "${PORT}" || echo false`
    url_result=`curl -s http://localhost:${PORT} || echo false `

    if [[ $proc_result == "false" || $port_result == "false" || $url_result == "false" ]];then
        echo "server not running"
    else
        echo "ok"
    fi
}

case $1 in
    start)
        start
        sleep 5
        check
        ;;

    stop)
        stop
        sleep 5
        check
        ;;
    restart)
        stop
        sleep 5
        start
        sleep 5
        check
        ;;
    check)
        check
        ;;
    *)
        echo "sh service.sh {start|stop|restart|check}"
        ;;
esac
  • service.sh脚本拷贝到测试节点:
[root@Devops6 devops6-maven-service_CD]#ansible 172.29.9.110 -m copy -a "src=service.sh  dest=/opt/service.sh"
172.29.9.110 | CHANGED => {
    "ansible_facts": {
        "discovered_interpreter_python": "/usr/bin/python"
    }, 
    "changed": true, 
    "checksum": "666b4746afbb9fa684f79a89102715906417c848", 
    "dest": "/opt/service.sh", 
    "gid": 0, 
    "group": "root", 
    "md5sum": "22868400cb2784f7c7bcf63f38a977fe", 
    "mode": "0644", 
    "owner": "root", 
    "size": 1367, 
    "src": "/root/.ansible/tmp/ansible-tmp-1688219597.85-41901-255991227715874/source", 
    "state": "file", 
    "uid": 0
}

然后启动程序:

给予脚本执行权限:

[root@devops-deploy1 opt]#ll
total 17284
-rw-r--r-- 1 root root 17690913 Jul  1 20:12 devops6-maven-service-9.9.9-b5cfb8ee.jar
-rw-r--r-- 1 root root     1367 Jul  1 21:53 service.sh
[root@devops-deploy1 opt]#chmod +x service.sh 

启动程序:

[root@devops-deploy1 opt]#sh service.sh start
ok
[root@devops-deploy1 opt]#ps -aux|grep java
root       7626 37.4  8.7 3202716 163300 pts/0  Sl   21:55   0:04 java -jar -Dserver.port=8080 devops6-maven-service-9.9.9-b5cfb8ee.jar
root       7674  0.0  0.0 112708   972 pts/0    R+   21:55   0:00 grep --color=auto java
[root@devops-deploy1 opt]#

image-20230701215558442

开始集成
  • 最终代码如下

image-20230702075024487

Deploy.groovy文件

package org.devops

//rollback
def AnsibleRollBack(){

    sh """
        # 停止服务
        ansible "${env.deployHosts}" -m shell -a "cd ${env.targetDir}/${env.projectName} ;source /etc/profile  && sh service.sh stop" -u root
        
        sleep 300
        # 清理和创建发布目录
        ansible "${env.deployHosts}" -m shell -a "rm -fr ${env.targetDir}/${env.projectName}/* &&  mkdir -p ${env.targetDir}/${env.projectName} || echo file is exists" 
        
        # 将备份目录内容复制到发布目录
        ansible "${env.deployHosts}" -m shell -a " mv ${env.targetDir}/${env.projectName}.bak/* ${env.targetDir}/${env.projectName}/ || echo file not exists"

        # 启动服务
        ansible "${env.deployHosts}" -m shell -a "cd ${env.targetDir}/${env.projectName} ;source /etc/profile  && sh service.sh  start" -u root

        # 检查服务 
        sleep 10
        ansible "${env.deployHosts}" -m shell -a "cd ${env.targetDir}/${env.projectName} ;source /etc/profile  && sh service.sh check" -u root

        """
}


//发布制品
def AnsibleDeploy(){
    //将主机写入清单文件
    sh "rm -fr hosts "
    for (host in "${env.deployHosts}".split(',')){
        sh " echo ${host} >> hosts"
    }


    // ansible 发布jar
    sh """
        # 主机连通性检测
        ansible "${env.deployHosts}" -m ping -i hosts 
        
        # 创建备份目录
        ansible "${env.deployHosts}" -m shell -a "mkdir -p ${env.targetDir}/${env.projectName}.bak || echo file is exists" 
        # 备份上次构建
        ansible "${env.deployHosts}" -m shell -a " mv ${env.targetDir}/${env.projectName}/* ${env.targetDir}/${env.projectName}.bak/ || echo file not exists"

        # 清理和创建发布目录
        ansible "${env.deployHosts}" -m shell -a "rm -fr ${env.targetDir}/${env.projectName}/* &&  mkdir -p ${env.targetDir}/${env.projectName} || echo file is exists" 
        # 复制app
        ansible "${env.deployHosts}" -m copy -a "src=${env.artifactName}  dest=${env.targetDir}/${env.projectName}/${env.artifactName}" 
    """
    
    // 发布脚本
    fileData = libraryResource 'scripts/service.sh'
    println(fileData)
    writeFile file: 'service.sh', text: fileData
    sh "ls -a ; cat service.sh "


    sh """
        # 修改变量
        sed -i 's#APPNAME=NULL#APPNAME=${env.projectName}#g' service.sh
        sed -i 's#VERSION=NULL#VERSION=${env.releaseVersion}#g' service.sh
        sed -i 's#PORT=NULL#PORT=${env.port}#g' service.sh

        # 复制脚本
        ansible "${env.deployHosts}" -m copy -a "src=service.sh  dest=${env.targetDir}/${env.projectName}/service.sh" 
        # 启动服务
        ansible "${env.deployHosts}" -m shell -a "cd ${env.targetDir}/${env.projectName} ;source /etc/profile && sh service.sh start" -u root

        # 检查服务 
        sleep 10
        ansible "${env.deployHosts}" -m shell -a "cd ${env.targetDir}/${env.projectName} ;source /etc/profile && sh service.sh  check" -u root
    """
}

cd.jenkinsfile文件

@Library("devops06@main") _ 

//import  src/org/devops/Gitlab.groovy
def mygit = new org.devops.Gitlab()
def mydeploy = new org.devops.Deploy()


//pipeline
pipeline{
    agent { label "build"}
    options {
        skipDefaultCheckout true
    }
    stages{
        stage("GetArtifact"){
            steps{
                script{
                    env.projectName = "${JOB_NAME}".split('_')[0] //devops6-maven-service
                    env.groupName = "${env.projectName}".split('-')[0]  //devops6
                    
                    projectID = mygit.GetProjectIDByName(env.projectName, env.groupName)
                    commitID = mygit.GetBranchCommitID("${projectID}", "${env.branchName}")
                    println(commitID)

                    appVersion = "${env.branchName}".split("-")[-1]  //9.9.9
                    println(appVersion)
                    // currentBuild.description = "Version: ${appVersion}-${commitID}"

                    currentBuild.displayName = "第${BUILD_NUMBER}次构建-${commitID}"
                    currentBuild.description = "构建分支名称:${env.branchName}"


                    //下载制品
                    //http://172.29.9.101:8081/repository/devops6/devops6-maven-service/6.1.1-b5cfb8ee/devops6-maven-service-6.1.1-b5cfb8ee.jar
                    repoUrl = "http://172.29.9.101:8081/repository/${env.groupName}"
                    env.artifactName = "${env.projectName}-${appVersion}-${commitID}.jar"
                    artifactUrl = "${repoUrl}/${env.projectName}/${appVersion}-${commitID}/${env.artifactName}"
                    sh "wget --no-verbose ${artifactUrl} && ls -l"

                    env.releaseVersion = "${appVersion}-${commitID}"
                
                }
            }
        }

        stage("Deploy"){
            steps{
                script{                    
                    mydeploy.AnsibleDeploy()
                }
            }
        }

        stage("RollBack"){
            input {
                message "是否进行回滚?"
                ok "Yes"
                submitter ""
                parameters {
                    choice choices: ['NO','YES'], name: 'OPS'
                }
            }
            steps {
                echo "OPS  ${OPS}, doing......."

                script{
                    if ("${OPS}" == "YES"){
                        mydeploy.AnsibleRollBack()
                    }
                }
                
            }
        }        

        

    }


}
  • 测试效果

执行CD流水线:

image-20230702075246458

运行成功:

image-20230702075357409

再看下2个节点的java运行情况:

image-20230702075656329

符合预期。

给gitlab上devops6-maven-service项目配置个健康检查端口
  • 默认这个生成的jar包启动后,是没配置健康检查端口的,我们的测试现象不明确

image-20230701192345639

我们来启动下服务:

image-20230701192548799

image-20230701192614173

  • 因此我们来改下这个java代码:

image-20230702113937993

BasicController.java

/*
 * Copyright 2013-2018 the original author or authors.
 *
 * Licensed under the Apache License, Version 2.0 (the "License");
 * you may not use this file except in compliance with the License.
 * You may obtain a copy of the License at
 *
 *      https://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */

package com.example.demo;

import org.springframework.stereotype.Controller;
import org.springframework.web.bind.annotation.ModelAttribute;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestParam;
import org.springframework.web.bind.annotation.ResponseBody;

/**
 * @author <a href="mailto:chenxilzx1@gmail.com">theonefx</a>
 */
@Controller
public class BasicController {

    // http://127.0.0.1:8080/hello?name=lisi
    @RequestMapping("/hello")
    @ResponseBody
    public String hello(@RequestParam(name = "name", defaultValue = "xyy") String name) {
        return "Hello RELEASE-10.1.0 " + name;
    }
}

然后打包,运行,观察效果:

image-20230702110042099

image-20230702110104946

  • 最后,将devops6-maven-serviceRELEASE-9.9.9/代码合并到main分支。

  • 打上tag

image-20230702112212540

image-20230702112219976

Step4: 回滚

image-20230701215632309

推荐第一种。

第二种方法会存在很多逻辑问题的。

  • 回滚代码见上述文件,这里测试下效果

1、直接发布版本方式来回滚

先运行CI流水线

image-20230702075835437

CI pipeline运行成功:

image-20230702075903928

image-20230702075919711

再运行CD:

image-20230702075951461

观察效果:

image-20230702080049756

image-20230702080104522

可以看到发布老版本程序成功。

2、使用回滚代码

注意:如果要回滚时,就需要跳过发布阶段,否则会有问题的,因此这里我给发布阶段加了一个判断选项。

发布1.1.1

image-20230702092448462

发布9.9.9

image-20230702092637660

回滚到1.1.1:

image-20230702092740901

image-20230702092800153

image-20230702092900471

符合预期。😘

扩展:参数动态获取实践

  • 需要安装active choices插件重启Jenkins服务器后再操作。

tstmp_20230702100125

根据不同的环境带出不同的机器
  • 效果

image-20230702101549409

image-20230702101941216

  • envName参数设置

image-20230702101647504

return ["dev", "uat", "stag", "prod"]
  • deployHosts参数设置

image-20230702101716888

if (envName.equals("dev")){
	return ["172.29.9.110,172.29.9.111"]
} else if (envName.equals("uat")){
	return ["172.29.9.120,172.29.9.121"]
}

image-20230702101741319

⚠️ 注意:记得删除前面定义好的envName和deployHosts选项参数。

  • 运行测试

image-20230702101549409

image-20230702101525323

image-20230702101609022

符合预期。😘

根据不同发布工具,动态展示主机参数

这个就不做演示了,和上面这个实践有冲突。

tstmp_20230702102119

  • 定义发布工具参数

tstmp_20230702102143

return ["ansible", "saltstack"]

单选类型

tstmp_20230702102211

  • 定义发布主机

tstmp_20230702102226

选择关联的参数,多个参数用逗号分割

tstmp_20230702102300

3、代码汇总

  • 本次实验代码

链接:https://pan.baidu.com/s/1mn1EX2oX0XRGO-IjohkyLA?pwd=0820
提取码:0820

2023.7.2-云主机模式持续部署-ci-cd-code

image-20230702194013010

  • 实验环境
gitlab-ce:15.0.3-ce.0
jenkins:2.346.3-2-lts-jdk11
sonarqube:9.9.0-community
nexus3:3.53.0

image-20230702194106832

  • ci-cd流水线

这2条流水线都是测试ok的。(以后就一直用这2条流水线来测试devops了)

image-20230702194138050

image-20230702194213612

image-20230702194227921

  • 仓库代码

gitlab仓库devops6-maven-service:RELEASE-9.9.9和main分支都是一样的代码。

image-20230702194255650

jenkins共享库代码:

image-20230702194401287

  • jenkins共享库代码汇总

image-20230702194944945

service.sh

#!/bin/bash

# sh service.sh anyops-devops-service 1.1.1 8091 start
APPNAME=NULL
VERSION=NULL
PORT=NULL

start(){
    port_result=`netstat -anlpt | grep "${PORT}" || echo false`

    if [[ $port_result == "false" ]];then
        nohup java -jar -Dserver.port=${PORT}  ${APPNAME}-${VERSION}.jar >${APPNAME}.log.txt 2>&1 &
    else
       stop
       sleep 5
       nohup java -jar -Dserver.port=${PORT}  ${APPNAME}-${VERSION}.jar >${APPNAME}.log.txt 2>&1 &
    fi
}


stop(){
    pid=`netstat -anlpt | grep "${PORT}" | awk '{print $NF}' | awk -F '/' '{print $1}' | head -1`
    kill -15 $pid
}


check(){
    proc_result=`ps aux | grep java | grep "${APPNAME}" | grep -v grep || echo false`
    port_result=`netstat -anlpt | grep "${PORT}" || echo false`
    url_result=`curl -s http://localhost:${PORT} || echo false `

    if [[ $proc_result == "false" || $port_result == "false" || $url_result == "false" ]];then
        echo "server not running"
    else
        echo "ok"
    fi
}

case $1 in
    start)
        start
        sleep 5
        check
        ;;

    stop)
        stop
        sleep 5
        check
        ;;
    restart)
        stop
        sleep 5
        start
        sleep 5
        check
        ;;
    check)
        check
        ;;
    *)
        echo "sh service.sh {start|stop|restart|check}"
        ;;
esac

Jenkinsfile

@Library("devops06@main") _

//import src/org/devops/xxx.groovy
def checkout = new org.devops.CheckOut()
def build = new org.devops.Build()
def sonar = new org.devops.Sonar()
def artifact = new org.devops.Artifact()
//def gitlab = new org.devops.GitLab()


//使用git 参数需要格式化
env.branchName = "${env.branchName}" - "origin/"
println(env.branchName)

pipeline {
    agent {label "build"}

    //跳过默认的代码检出功能
    options {
        skipDefaultCheckout true
    }    
    

    stages{
        stage("CheckOut"){
            steps{
                script{
                    checkout.CheckOut()
                    
                    //获取commitID
                    env.commitID = checkout.GetCommitID()
                    println("commitID: ${env.commitID}")

                    // Jenkins构建显示信息
                    currentBuild.displayName = "第${BUILD_NUMBER}次构建-${env.commitID}"
                    currentBuild.description = "构建分支名称:${env.branchName}"
                    //currentBuild.description = "Trigger by user jenkins \n branch: ${env.branchName}"
                }
            }
        }

        stage("Build"){
            steps{
                script{
                    build.Build()
                }
            }

        }        

        stage("CodeScan"){
            // 是否跳过代码扫描?
            when {
                environment name: 'skipSonar', value: 'false'
            }
            
            steps{
                script{
                    sonar.SonarScannerByPlugin()        

                }
            }
        }

         stage("PushArtifact"){
            steps{
                script{
                    //PushArtifactByPlugin()
                    //PushArtifactByPluginPOM()

                    // init package info
                    appName = "${JOB_NAME}".split('_')[0] //devops6-maven-service_CI
                    repoName = appName.split('-')[0]   //devops6
                    appVersion = "${env.branchName}".split("-")[-1] // RELEASE-1.1.1   1.1.1
                    appVersion = "${appVersion}-${env.commitID}"
                    targetDir="${appName}/${appVersion}"
                    

                    // 通过pom文件获取包名称
                    POM = readMavenPom file: 'pom.xml'
                    env.artifactId = "${POM.artifactId}"
                    env.packaging = "${POM.packaging}"
                    env.groupId = "${POM.groupId}"
                    env.art_version = "${POM.version}"
                    sourcePkgName = "${env.artifactId}-${env.art_version}.${env.packaging}"
                    
                    pkgPath = "target"
                    targetPkgName = "${appName}-${appVersion}.${env.packaging}"
                    artifact.PushNexusArtifact(repoName, targetDir, pkgPath, sourcePkgName,targetPkgName)
                }
            }
    
        }       

    }
}

/*
//通过nexus api上传制品--综合实践
def PushNexusArtifact(repoId, targetDir, pkgPath, sourcePkgName,targetPkgName){
    //nexus api 
    withCredentials([usernamePassword(credentialsId: '3404937d-89e3-4699-88cf-c4bd299094ad', \
                                    passwordVariable: 'PASSWD', 
                                    usernameVariable: 'USERNAME')]) {
        sh """
            curl -X 'POST' \
              "http://172.29.9.101:8081/service/rest/v1/components?repository=${repoId}" \
              -H 'accept: application/json' \
              -H 'Content-Type: multipart/form-data' \
              -F "raw.directory=${targetDir}" \
              -F "raw.asset1=@${pkgPath}/${sourcePkgName};type=application/java-archive" \
              -F "raw.asset1.filename=${targetPkgName}" \
              -u ${USERNAME}:${PASSWD}
        """
    }
}
*/

cd.jenkinsfile

@Library("devops06@main") _ 

//import  src/org/devops/Gitlab.groovy
def mygit = new org.devops.Gitlab()
def mydeploy = new org.devops.Deploy()


//pipeline
pipeline{
    agent { label "build"}
    options {
        skipDefaultCheckout true
    }
    stages{
        stage("GetArtifact"){
            steps{
                script{
                    env.projectName = "${JOB_NAME}".split('_')[0] //devops6-maven-service
                    env.groupName = "${env.projectName}".split('-')[0]  //devops6
                    
                    projectID = mygit.GetProjectIDByName(env.projectName, env.groupName)
                    commitID = mygit.GetBranchCommitID("${projectID}", "${env.branchName}")
                    println(commitID)

                    appVersion = "${env.branchName}".split("-")[-1]  //9.9.9
                    println(appVersion)
                    // currentBuild.description = "Version: ${appVersion}-${commitID}"

                    currentBuild.displayName = "第${BUILD_NUMBER}次构建-${commitID}"
                    currentBuild.description = "构建分支名称:${env.branchName}"


                    //下载制品
                    //http://172.29.9.101:8081/repository/devops6/devops6-maven-service/6.1.1-b5cfb8ee/devops6-maven-service-6.1.1-b5cfb8ee.jar
                    repoUrl = "http://172.29.9.101:8081/repository/${env.groupName}"
                    env.artifactName = "${env.projectName}-${appVersion}-${commitID}.jar"
                    artifactUrl = "${repoUrl}/${env.projectName}/${appVersion}-${commitID}/${env.artifactName}"
                    sh "wget --no-verbose ${artifactUrl} && ls -l"

                    env.releaseVersion = "${appVersion}-${commitID}"
                
                }
            }
        }

        stage("Deploy"){

            // 是否跳过发布?
            when {
                environment name: 'skipDeploy', value: 'false'
            }
            
            steps{
                script{                    
                    mydeploy.AnsibleDeploy()
                }
            }
        }

        stage("RollBack"){
            input {
                message "是否进行回滚?"
                ok "Yes"
                submitter ""
                parameters {
                    choice choices: ['NO','YES'], name: 'OPS'
                }
            }
            steps {
                echo "OPS  ${OPS}, doing......."

                script{
                    if ("${OPS}" == "YES"){
                        mydeploy.AnsibleRollBack()
                    }
                }
                
            }
        }        

        

    }


}

Deploy.groovy

package org.devops

//rollback
def AnsibleRollBack(){

    sh """
        # 停止服务
        ansible "${env.deployHosts}" -m shell -a "cd ${env.targetDir}/${env.projectName} ;source /etc/profile  && sh service.sh stop" -u root
        
        sleep 20
        # 清理和创建发布目录
        ansible "${env.deployHosts}" -m shell -a "rm -fr ${env.targetDir}/${env.projectName}/* &&  mkdir -p ${env.targetDir}/${env.projectName} || echo file is exists" 
        
        # 将备份目录内容复制到发布目录
        ansible "${env.deployHosts}" -m shell -a " mv ${env.targetDir}/${env.projectName}.bak/* ${env.targetDir}/${env.projectName}/ || echo file not exists"

        # 启动服务
        ansible "${env.deployHosts}" -m shell -a "cd ${env.targetDir}/${env.projectName} ;source /etc/profile  && sh service.sh  start" -u root

        # 检查服务 
        sleep 10
        ansible "${env.deployHosts}" -m shell -a "cd ${env.targetDir}/${env.projectName} ;source /etc/profile  && sh service.sh check" -u root

        """
}


//发布制品
def AnsibleDeploy(){
    //将主机写入清单文件
    sh "rm -fr hosts "
    for (host in "${env.deployHosts}".split(',')){
        sh " echo ${host} >> hosts"
    }


    // ansible 发布jar
    sh """
        # 主机连通性检测
        ansible "${env.deployHosts}" -m ping -i hosts 
        
        # 创建备份目录
        ansible "${env.deployHosts}" -m shell -a "mkdir -p ${env.targetDir}/${env.projectName}.bak || echo file is exists" 
        # 备份上次构建
        ansible "${env.deployHosts}" -m shell -a " mv ${env.targetDir}/${env.projectName}/* ${env.targetDir}/${env.projectName}.bak/ || echo file not exists"

        # 清理和创建发布目录
        ansible "${env.deployHosts}" -m shell -a "rm -fr ${env.targetDir}/${env.projectName}/* &&  mkdir -p ${env.targetDir}/${env.projectName} || echo file is exists" 
        # 复制app
        ansible "${env.deployHosts}" -m copy -a "src=${env.artifactName}  dest=${env.targetDir}/${env.projectName}/${env.artifactName}" 
    """
    
    // 发布脚本
    fileData = libraryResource 'scripts/service.sh'
    println(fileData)
    writeFile file: 'service.sh', text: fileData
    sh "ls -a ; cat service.sh "


    sh """
        # 修改变量
        sed -i 's#APPNAME=NULL#APPNAME=${env.projectName}#g' service.sh
        sed -i 's#VERSION=NULL#VERSION=${env.releaseVersion}#g' service.sh
        sed -i 's#PORT=NULL#PORT=${env.port}#g' service.sh

        # 复制脚本
        ansible "${env.deployHosts}" -m copy -a "src=service.sh  dest=${env.targetDir}/${env.projectName}/service.sh" 
        # 启动服务
        ansible "${env.deployHosts}" -m shell -a "cd ${env.targetDir}/${env.projectName} ;source /etc/profile && sh service.sh start" -u root

        # 检查服务 
        sleep 10
        ansible "${env.deployHosts}" -m shell -a "cd ${env.targetDir}/${env.projectName} ;source /etc/profile && sh service.sh  check" -u root
    """
}


Gitlab.groovy

package org.devops

//发起HTTP请求
//调用gitlab api
def HttpReq(method, apiUrl){

    withCredentials([string(credentialsId: '5782c77d-ce9d-44e5-b9ba-1ba2097fc31d', variable: 'gitlabtoken')]) {
        response = sh  returnStdout: true, 
        script: """ 
        curl --location --request ${method} \
        http://172.29.9.101:8076/api/v4/${apiUrl} \
        --header "PRIVATE-TOKEN: ${gitlabtoken}"
        """
    }
    response = readJSON text: response - "\n" //json数据的读取方式
    return response
}

//获取ProjectID
def GetProjectIDByName(projectName, groupName){
    apiUrl = "projects?search=${projectName}"
    response = HttpReq("GET", apiUrl)
    if (response != []){
        for (p in response) {
            if (p["namespace"]["name"] == groupName){
                return response[0]["id"]
            }
        }
    }
}

//获取分支CommitID
def GetBranchCommitID(projectID, branchName){
    apiUrl = "projects/${projectID}/repository/branches/${branchName}"
    response = HttpReq("GET", apiUrl)
    return response.commit.short_id
}
  • CI页面参数如下

image-20230702200353487

image-20230702200406135

image-20230702200416790

image-20230702200427680

image-20230702200442435

image-20230702200459360

image-20230702200514574

image-20230702200521907

  • CD页面参数

image-20230702200558751

image-20230702200611962

image-20230702200620866

image-20230702200635806

image-20230702200646089

image-20230702200702612

image-20230702200714662

image-20230702200730449

image-20230702200739069

关于我

我的博客主旨:

  • 排版美观,语言精炼;
  • 文档即手册,步骤明细,拒绝埋坑,提供源码;
  • 本人实战文档都是亲测成功的,各位小伙伴在实际操作过程中如有什么疑问,可随时联系本人帮您解决问题,让我们一起进步!

🍀 微信二维码
x2675263825 (舍得), qq:2675263825。

image-20230107215114763

🍀 微信公众号
《云原生架构师实战》

image-20230107215126971

🍀 语雀

https://www.yuque.com/xyy-onlyone

image-20230624093747671

🍀 csdn
https://blog.csdn.net/weixin_39246554?spm=1010.2135.3001.5421

image-20230107215149885

🍀 知乎
https://www.zhihu.com/people/foryouone

image-20230107215203185

最后

好了,关于本次就到这里了,感谢大家阅读,最后祝大家生活快乐,每天都过的有意义哦,我们下期见!

image-20230702201359916
response = HttpReq(“GET”, apiUrl)
if (response != []){
for (p in response) {
if (p[“namespace”][“name”] == groupName){
return response[0][“id”]
}
}
}
}

//获取分支CommitID
def GetBranchCommitID(projectID, branchName){
apiUrl = “projects/ p r o j e c t I D / r e p o s i t o r y / b r a n c h e s / {projectID}/repository/branches/ projectID/repository/branches/{branchName}”
response = HttpReq(“GET”, apiUrl)
return response.commit.short_id
}




- CI页面参数如下

[外链图片转存中...(img-QfPAfydf-1688300616371)]

[外链图片转存中...(img-W6JQCH3L-1688300616371)]

[外链图片转存中...(img-JyUYqNZD-1688300616372)]

[外链图片转存中...(img-cDeigiAQ-1688300616372)]

[外链图片转存中...(img-bDEQtVsW-1688300616372)]

[外链图片转存中...(img-fvknHcLW-1688300616372)]

[外链图片转存中...(img-qyBydz95-1688300616373)]

[外链图片转存中...(img-a14KH4NR-1688300616373)]



- CD页面参数

[外链图片转存中...(img-1O4fqE3z-1688300616373)]

[外链图片转存中...(img-jjzRnpju-1688300616374)]

[外链图片转存中...(img-ZK2ULdpb-1688300616374)]

[外链图片转存中...(img-mveq9EUt-1688300616374)]

[外链图片转存中...(img-y9sgD3ZV-1688300616375)]

[外链图片转存中...(img-uwlCPq5f-1688300616375)]

[外链图片转存中...(img-Ws9rX59L-1688300616375)]

[外链图片转存中...(img-GtRKkUw5-1688300616375)]

[外链图片转存中...(img-1DUDIHmE-1688300616376)]

## 关于我

我的博客主旨:

- 排版美观,语言精炼;
- 文档即手册,步骤明细,拒绝埋坑,提供源码;
- 本人实战文档都是亲测成功的,各位小伙伴在实际操作过程中如有什么疑问,可随时联系本人帮您解决问题,让我们一起进步!



🍀 微信二维码
x2675263825 (舍得), qq:2675263825。

[外链图片转存中...(img-p0fEeEL4-1688300616376)]



🍀 微信公众号
《云原生架构师实战》

[外链图片转存中...(img-O6MQ006k-1688300616376)]



🍀 语雀

https://www.yuque.com/xyy-onlyone

[外链图片转存中...(img-5TROvRge-1688300616377)]



🍀 csdn
[https://blog.csdn.net/weixin_39246554?spm=1010.2135.3001.5421](https://blog.csdn.net/weixin_39246554?spm=1010.2135.3001.5421)

[外链图片转存中...(img-vUzjbf7l-1688300616379)]



🍀 知乎
[https://www.zhihu.com/people/foryouone](https://www.zhihu.com/people/foryouone)

[外链图片转存中...(img-EYtkEa5o-1688300616379)]

## 最后

好了,关于本次就到这里了,感谢大家阅读,最后祝大家生活快乐,每天都过的有意义哦,我们下期见!

[外链图片转存中...(img-MUi6bZl1-1688300616379)]
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值