什么是GitOps
GitOps既Git + Ops
Git：分布式版本控制
Ops：来自DevOps的概念，Development Operations 开发运营（运维）
GitOps：基于分布式版本控制软件Git的开发运营（运维）
具体可以参考一下这个B站视频，介绍的很不错：
【GitOps是什么？它是如何工作的？为什么它如此有用？中国DevOps社区】
GitOps如何选择工具
GitOps首先最重要的一定是要有Git工具，这个大家在开发过程中肯定不陌生，GitHub、Gitlab等等应该是大家常用的。公司内部作为代码安全性考虑，一般会选择把代码放在私有化部署的Gitlab上，当然也可以有别的选择，此处我们不做展开。
有了GItlab作为版本控制了，我们还需要CI/CD的工具，CI的工具有Jenkins、Tekton等第三方的，也可以用Gitlab CI、GitHub Action等工具自带的，基于广泛适用性以及高度自由的可配置性考虑，我选择使用Jenkins。
对于CD工具，我们可以选择通用的Jenkins、Tekton等，但是在K8S环境下，专用的CD工具还要数ArgoCD和FluxCD，对于现阶段而言，个人可能更倾向于ArgoCD，它不仅支持多租户、UI、多对多等功能，而且相比较于jenkins对于配置的push，它采用pull的方式，在CD的过程中能够检查最终的部署状态。其具体差别，大家可以自行查阅。
具体的工具部署不是本文的重点，在此不做赘述。
流水线的设计
工具选择完毕，我们需要考虑怎么设计我们的流水线，以使各种工具协同工作，将整个编译、测试、部署等工作自动化。
我在这里画了一个简单的草图，来展示我的思路：
设计思路
-
当前GitOps的最佳实践就是把代码和配置分别管理。假设我们需要开发一个服务，我们暂且称之为demo，那么我们首选在gitlab中设计两个项目：代码项目–app和配置项目–helm。
-
app项目的代码分支配置我们不做讨论，这是研发的事儿，但是配置仓库事关部署，我们这边决定采用Gitlab Flow，既分支对应环境的方式，我们可以依据当前场景先定义dev和test两个分支
-
然后我们在harbor中有两个project，分别为demo-dev、demo-test，分别对应开发环境的K8S集群和测试环境的K8S集群，以实现镜像与环境的对应关系。
-
ArgoCD根据配置仓库helm的不同分支对不同环境的集群配置进行同步，从harbor对应的project中拉取对应的镜像。
说明：此处我们使用helm，大家也可以选择kustomize，具体大家根据实际情况自行选择。
在实践的过程中我们也讨论过一个问题，就是在新的迭代期间，发现上个迭代的线上出现BUG怎么办，最后讨论的暂定结果是修复BUG的时候，单独切换流水线触发的分支到BUG修复分支上，暂时没有发现其他更好的方式，欢迎大家分享。
流水线实现
-
研发在本地开发完成代码后，push到对应的gitlab，此处为dev分支，push触发Jenkins流水线，执行CI流程，包括单元测试、代码检测、编译打包、推送、并对配置仓库的dev分支进行更新，在这个过程中，会生成image并推送到harbor的demo-dev项目中，并将新生成的image的tag更新到helm的dev分支。整个过程可以增加多步审核，当然，开发环境的部署时研发人员的自主行为，一般不需要做审核操作。
pipeline { agent { node { label 'maven' } } environment { DOCKER_CREDENTIAL_ID = 'harbor-test' //在K8S集群内配置的secrect GITLAB_CREDENTIAL_ID = 'gitlab' //在K8S集群内配置的secrect REGISTRY = 'harbor-test.example.com' DOCKERHUB_NAMESPACE = 'demo' APP_NAME = 'demo' SONAR_CREDENTIAL_ID = 'sonar-token' //在K8S集群内配置的secrect } stages { stage('unit test') { steps { container('maven') { sh 'mvn clean test' } } } stage('sonarqube analysis') { steps { container('maven') { withCredentials([string(credentialsId: "$SONAR_CREDENTIAL_ID", variable: 'SONAR_TOKEN')]) { withSonarQubeEnv('sonar') { sh 'mvn sonar:sonar -Dsonar.login=$SONAR_TOKEN' } } timeout(time: 1, unit: 'HOURS') { waitForQualityGate abortPipeline: true } } } } stage('build & push') { when { branch 'dev' //可以根据多分支配置不通的执行条件，具体情况根据实际需求制定 } steps { container('maven') { script{ env.ctag=sh(script: 'git log --pretty=format:"%H" -n 1', returnStdout: true).trim() env.IMAGE_TAG=sh(script: 'echo ${ctag:0:10}', returnStdout: true).trim() } sh 'mvn clean package -DskipTests' sh 'docker build -f Dockerfile -t $REGISTRY/$DOCKERHUB_NAMESPACE/$APP_NAME:$IMAGE_TAG .' withCredentials([usernamePassword(passwordVariable: 'DOCKER_PASSWORD', usernameVariable: 'DOCKER_USERNAME', credentialsId: "$DOCKER_CREDENTIAL_ID",)]) { sh 'echo "$DOCKER_PASSWORD" | docker login $REGISTRY -u "$DOCKER_USERNAME" --password-stdin' sh 'docker push $REGISTRY/$DOCKERHUB_NAMESPACE/$APP_NAME:$IMAGE_TAG' } } } } stage('update docker tag') { when { branch 'dev' //可以根据多分支配置不通的执行条件，具体情况根据实际需求制定 } environment { BUILD_USER = 'wangjb' BUILD_USER_EMAIL = 'wangjb@jovision.com' YAML_REPO_URL='http://${username}:${password}@gitlab.example.com/cloud-native/helm-demo.git' } steps { withCredentials([usernamePassword(passwordVariable : 'password' ,usernameVariable : 'username' ,credentialsId : "$GITLAB_CREDENTIAL_ID" ,)]) { sh """ git config --global user.name "$BUILD_USER" git config --global user.email "$BUILD_USER_EMAIL" git clone ${YAML_REPO_URL} && cd helm-demo sed -i 's/imagetag:.*/imagetag: "'$IMAGE_TAG'"/' demo/values.yaml git add -A && git commit -m "update tag: ${IMAGE_TAG}" && git push ${YAML_REPO_URL} """ } } } } } -
ArgoCD检测到helm的dev分支配置与cluster-dev不一致，进行sync操作，从harbor的demo-dev拉取指定的image:tag1。
-
经过dev环境验证后，将helm的dev分支merge到test分支，同时触发流水线将harbor的demo-dev的相应image复制到demo-test，ArgoCD检测配置并进行自动同步，并从demo-test拉取image:tag1。
pipeline {
agent any
environment {
DOCKER_CREDENTIAL_ID = 'harbor-test' //在K8S集群内配置的secrect
DOCKER_CREDENTIAL_ID_PROD = 'harbor-prod' //在K8S集群内配置的secrect
GITHUB_CREDENTIAL_ID = 'gitlab' //在K8S集群内配置的secrect
REGISTRY_TEST = 'http://harbor-test.example.com'
REGISTRY_PROD = 'http://harbor.example.com'
PROJECT_NAME = "demo-test"
REPOSITORY_NAME = 'demo' //The name of the repository. If it contains slash, encode it with URL encoding. e.g. a/b -> a%252Fb
ARTIFACT_FROM = "demo/demo"
}
stages {
stage('镜像同步') {
when {
branch 'prod'
}
steps {
container('base') {
script{
env.ARTIFACT_TAG=sh(script: 'cat demo/values.yaml | grep "imagetag:" | awk -F "\\"" \'{print $2}\'', returnStdout: true).trim()
}
withCredentials([usernamePassword(passwordVariable: 'DOCKER_PASSWORD', usernameVariable: 'DOCKER_USERNAME', credentialsId: "$DOCKER_CREDENTIAL_ID",)]) {
sh '''
echo "COPY ARTIFACT"
result=`curl -X 'POST' -u $DOCKER_USERNAME:$DOCKER_PASSWORD -i \
"$REGISTRY_TEST/api/v2.0/projects/$PROJECT_NAME/repositories/$REPOSITORY_NAME/artifacts?from=$ARTIFACT_FROM:$ARTIFACT_TAG" \
-H 'accept: application/json' | grep "HTTP/1.1 201 Created" | grep -v grep | wc -l`
if [ $result -eq 1 ];then
echo "The artifact copy is successful!"
else
echo "The artifact copy is faild!"
exit 1
fi
'''
}
}
}
}
}
}
- 如果是需要修改helm的其他配置，则可以直接拉取helm-dev、修改后直接推送，之后接merge等操作部署到其他环境即可。
- 多环境（pre-prod、prod等）按同理进行拓展即可
消息通知
Jenkins因为可以使用shell和groovy脚本，因此可以直接调用webhook对接通知接收工具如飞书、钉钉等，此处不做展开。
ArgoCD可以监控应用状态变化，但是它本身却没有能力将变化发送通知，这时候我们就可以使用argocd-notifications
argocd-notifications本身是支持Slack, SMTP, Opsgenie, Telegram 和其他任何使用webhooks的应用。因此我们可以通过使用它的webhook来发送飞书或者钉钉消息。
安装部署
下载并进行修改
首先将配置文件下载到本地：
wget https://raw.githubusercontent.com/argoproj-labs/argocd-notifications/master/manifests/install.yaml
然后对configmap部分进行修改
apiVersion: v1
kind: ConfigMap
metadata:
creationTimestamp: null
name: argocd-notifications-cm
data:
service.webhook.feishu: |
url: https://open.feishu.cn/open-apis/bot/v2/hook/******-******-****** #飞书自定义机器人的webhook
headers:
- name: Content-Type
value: application/json
context: |
argocdUrl: https://www.argocd.com #argocd的web页面，用于消息通知中直接跳转
clusterUrl: http://www.k8scluster.com #K8Sweb地址，用于在消息中直接跳转到集群中的相关页面
template.app-sync-healthy: |
webhook:
feishu:
method: POST
body: |
{
"msg_type": "interactive",
"card": {
"elements": [
{
"tag": "column_set",
"flex_mode": "none",
"background_style": "default",
"columns": [
{
"tag": "column",
"width": "weighted",
"weight": 1,
"vertical_align": "top",
"elements": [
{
"tag": "div",
"text": {
"content": "**⚙ app名称：**\n {{.app.metadata.name}}",
"tag": "lark_md"
}
}
]
},
{
"tag": "column",
"width": "weighted",
"weight": 1,
"vertical_align": "top",
"elements": [
{
"tag": "div",
"text": {
"content": "**⏱ 时间：**\n {{ (call .time.Parse .app.status.operationState.startedAt).Local.Format "2006-01-02 15:04:05(Z07:00)" }}",
"tag": "lark_md"
}
}
]
}
]
},
{
"tag": "column_set",
"flex_mode": "none",
"background_style": "default",
"columns": [
{
"tag": "column",
"width": "weighted",
"weight": 1,
"vertical_align": "top",
"elements": [
{
"tag": "div",
"text": {
"content": "**✔ app同步状态:**\n {{.app.status.operationState.phase}}",
"tag": "lark_md"
}
}
]
},
{
"tag": "column",
"width": "weighted",
"weight": 1,
"vertical_align": "top",
"elements": [
{
"tag": "div",
"text": {
"content": "**💚 app状态:**\n {{.app.status.health.status}}",
"tag": "lark_md"
}
}
]
}
]
},
{
"tag": "column_set",
"flex_mode": "none",
"background_style": "default",
"columns": [
{
"tag": "column",
"width": "weighted",
"weight": 1,
"vertical_align": "top",
"elements": [
{
"tag": "div",
"text": {
"content": "**☸ 所属集群:**\n {{ .app.spec.destination.name}}",
"tag": "lark_md"
}
}
]
},
{
"tag": "column",
"width": "weighted",
"weight": 1,
"vertical_align": "top",
"elements": [
{
"tag": "div",
"text": {
"content": "**⚪ 集群NS:**\n {{ .app.spec.destination.namespace}}",
"tag": "lark_md"
}
}
]
}
]
},
{
"tag": "action",
"actions": [
{
"tag": "button",
"text": {
"tag": "plain_text",
"content": "跳转ArgoCD"
},
"type": "primary",
"multi_url": {
"url": "{{.context.argocdUrl}}/applications/{{.app.metadata.name}}?operation=true",
"pc_url": "",
"android_url": "",
"ios_url": ""
}
},
{
"tag": "button",
"text": {
"tag": "plain_text",
"content": "跳转Deployment"
},
"type": "default",
"multi_url": {
"url": "{{.context.clusterUrl}}/awbaby/clusters/{{.app.spec.destination.name}}/projects/{{.app.metadata.name}}/deployments",
"pc_url": "",
"android_url": "",
"ios_url": ""
}
}
]
},
{
"tag": "hr"
},
{
"tag": "div",
"fields": [
{
"is_short": true,
"text": {
"tag": "lark_md",
"content": "**Gitlab CommitID：**\n{{ .app.status.operationState.operation.sync.revision}}\n"
}
}
]
}
],
"header": {
"template": "green",
"title": {
"content": "ArgoCD同步通知",
"tag": "plain_text"
}
}
}
}
template.app-sync-unhealthy: |
webhook:
feishu:
method: POST
body: |
{
"msg_type": "interactive",
"card": {
"elements": [
{
"tag": "column_set",
"flex_mode": "none",
"background_style": "default",
"columns": [
{
"tag": "column",
"width": "weighted",
"weight": 1,
"vertical_align": "top",
"elements": [
{
"tag": "div",
"text": {
"content": "**⚙ app名称：**\n {{.app.metadata.name}}",
"tag": "lark_md"
}
}
]
},
{
"tag": "column",
"width": "weighted",
"weight": 1,
"vertical_align": "top",
"elements": [
{
"tag": "div",
"text": {
"content": "**⏱ 时间：**\n {{ (call .time.Parse .app.status.operationState.startedAt).Local.Format "2006-01-02 15:04:05(Z07:00)" }}",
"tag": "lark_md"
}
}
]
}
]
},
{
"tag": "column_set",
"flex_mode": "none",
"background_style": "default",
"columns": [
{
"tag": "column",
"width": "weighted",
"weight": 1,
"vertical_align": "top",
"elements": [
{
"tag": "div",
"text": {
"content": "**✔ app同步状态:**\n {{.app.status.operationState.phase}}",
"tag": "lark_md"
}
}
]
},
{
"tag": "column",
"width": "weighted",
"weight": 1,
"vertical_align": "top",
"elements": [
{
"tag": "div",
"text": {
"content": "**💔 app状态:**\n {{.app.status.health.status}}",
"tag": "lark_md"
}
}
]
}
]
},
{
"tag": "column_set",
"flex_mode": "none",
"background_style": "default",
"columns": [
{
"tag": "column",
"width": "weighted",
"weight": 1,
"vertical_align": "top",
"elements": [
{
"tag": "div",
"text": {
"content": "**☸ 所属集群:**\n {{ .app.spec.destination.name}}",
"tag": "lark_md"
}
}
]
},
{
"tag": "column",
"width": "weighted",
"weight": 1,
"vertical_align": "top",
"elements": [
{
"tag": "div",
"text": {
"content": "**⚪ 集群NS:**\n {{ .app.spec.destination.namespace}}",
"tag": "lark_md"
}
}
]
}
]
},
{
"tag": "action",
"actions": [
{
"tag": "button",
"text": {
"tag": "plain_text",
"content": "跳转ArgoCD"
},
"type": "primary",
"multi_url": {
"url": "{{.context.argocdUrl}}/applications/{{.app.metadata.name}}?operation=true",
"pc_url": "",
"android_url": "",
"ios_url": ""
}
},
{
"tag": "button",
"text": {
"tag": "plain_text",
"content": "跳转Deployment"
},
"type": "default",
"multi_url": {
"url": "{{.context.clusterUrl}}/awbaby/clusters/{{.app.spec.destination.name}}/projects/{{.app.metadata.name}}/deployments",
"pc_url": "",
"android_url": "",
"ios_url": ""
}
}
]
},
{
"tag": "hr"
},
{
"tag": "div",
"fields": [
{
"is_short": true,
"text": {
"tag": "lark_md",
"content": "**Gitlab CommitID：**\n{{ .app.status.operationState.operation.sync.revision}}\n"
}
}
]
}
],
"header": {
"template": "red",
"title": {
"content": "ArgoCD同步通知",
"tag": "plain_text"
}
}
}
}
template.app-sync-err-unhealthy: |
webhook:
feishu:
method: POST
body: |
{
...
}
template.app-sync-running: |
webhook:
feishu:
method: POST
body: |
{
...
}
template.app-sync-change-succ: |
webhook:
feishu:
method: POST
body: |
{
...
}
trigger.on-deployed: |
- description: Application is synced and healthy. Triggered once per commit.
oncePer: app.status.operationState.operation.sync.revision
send: [app-sync-healthy] # template names
# trigger condition
when: app.status.operationState.phase in ['Succeeded'] and app.status.health.status == 'Healthy'
trigger.on-health-degraded: |
- description: Application has degraded
send: [app-sync-unhealthy]
when: app.status.operationState.phase in ['Succeeded'] and app.status.health.status in ['Degraded','Unknown']
trigger.on-sync-failed: |
- description: Application syncing has failed
send: [app-sync-err-unhealthy] # template names
when: app.status.operationState.phase in ['Error','Failed']
trigger.on-sync-running: |
- description: Application is being synced
send: [app-sync-running] # template names
when: app.status.operationState.phase in ['Running']
trigger.on-sync-status-unknown: |
- description: Application status is 'Unknown,Degraded' and Application syncing has failed
send: [app-sync-err-unhealthy] # template names
when: app.status.operationState.phase in ['Error','Failed'] and app.status.sync.status in ['Degraded','Unknown']
trigger.on-sync-succeeded: |
- description: Application syncing has succeeded
send: [app-sync-change-succ] # template names
when: app.status.operationState.phase in ['Succeeded']
subscriptions: |
- recipients: [feishu] # 可能有bug，正常应该是webhook:feishu
triggers: [on-sync-running, on-deployed, on-sync-failed, on-sync-succeeded,on-health-degraded]
-
template.app-sync-healthy、app-sync-err-unhealthy、app-sync-running等等都是可以根据自己的需要自行制定消息模板，我这个是基于飞书卡片通知的，具体模板制定请参考飞书官方，同样，也可以制定钉钉的消息模板，方式类似，具体可以参考钉钉官方文档
-
trigger.on-deployed、trigger.on-health-degraded、trigger.on-sync-failed这些是触发器，即在满足什么条件的情况下，会触发使用哪个模板发送消息
subscriptions是指定启用哪些触发器，还有更多详细功能，建议参考官方文档进行合适的定制开发。
执行安装
使用kubectl部署到argocd的namespace中
kubectl apply -f install.yaml -n argocd
当pod就绪后，就可以开始测试了，当然在执行项目实际测试之前，可以先手动触发测试
kubectl exec -it argocd-notifications-controller-5685f9ccbf-n7plj -n argocd -- /app/argocd-notifications template notify app-sync-err-unhealthy demo --recipient feishu
小技巧分享
最后推荐一个网站：https://unicode.org/emoji/charts/full-emoji-list.html
在shell中有些情况需要使用表情符的时候，可以参考这个代码，我飞书通知中使用的某些表情符号就是在这找的。
使用方法，比如说要显示🧡，在vim中使用大写锁定键，然后按ctrl+v，在^状态下，分别输入U1F9E1，不需要写输入那个“+”，然后ECS退出编辑或者回车都可以显示出符号，但是这个符号在shell中是没有颜色的，在跟随其他的消息，如飞书通知等输出后，就是我们想要的符号了。
647
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
