什么是GitOps
GitOps既Git + Ops
Git:分布式版本控制
Ops:来自DevOps的概念,Development Operations 开发运营(运维)
GitOps:基于分布式版本控制软件Git的开发运营(运维)
具体可以参考一下这个B站视频,介绍的很不错:
【GitOps是什么?它是如何工作的?为什么它如此有用?中国DevOps社区】
GitOps如何选择工具
GitOps首先最重要的一定是要有Git工具,这个大家在开发过程中肯定不陌生,GitHub、Gitlab等等应该是大家常用的。公司内部作为代码安全性考虑,一般会选择把代码放在私有化部署的Gitlab上,当然也可以有别的选择,此处我们不做展开。
有了GItlab作为版本控制了,我们还需要CI/CD的工具,CI的工具有Jenkins、Tekton等第三方的,也可以用Gitlab CI、GitHub Action等工具自带的,基于广泛适用性以及高度自由的可配置性考虑,我选择使用Jenkins。
对于CD工具,我们可以选择通用的Jenkins、Tekton等,但是在K8S环境下,专用的CD工具还要数ArgoCD和FluxCD,对于现阶段而言,个人可能更倾向于ArgoCD,它不仅支持多租户、UI、多对多等功能,而且相比较于jenkins对于配置的push,它采用pull的方式,在CD的过程中能够检查最终的部署状态。其具体差别,大家可以自行查阅。
具体的工具部署不是本文的重点,在此不做赘述。
流水线的设计
工具选择完毕,我们需要考虑怎么设计我们的流水线,以使各种工具协同工作,将整个编译、测试、部署等工作自动化。
我在这里画了一个简单的草图,来展示我的思路:
设计思路
-
当前GitOps的最佳实践就是把代码和配置分别管理。假设我们需要开发一个服务,我们暂且称之为demo,那么我们首选在gitlab中设计两个项目:代码项目–app和配置项目–helm。
-
app项目的代码分支配置我们不做讨论,这是研发的事儿,但是配置仓库事关部署,我们这边决定采用Gitlab Flow,既分支对应环境的方式,我们可以依据当前场景先定义dev和test两个分支
-
然后我们在harbor中有两个project,分别为demo-dev、demo-test,分别对应开发环境的K8S集群和测试环境的K8S集群,以实现镜像与环境的对应关系。
-
ArgoCD根据配置仓库helm的不同分支对不同环境的集群配置进行同步,从harbor对应的project中拉取对应的镜像。
说明:此处我们使用helm,大家也可以选择kustomize,具体大家根据实际情况自行选择。
在实践的过程中我们也讨论过一个问题,就是在新的迭代期间,发现上个迭代的线上出现BUG怎么办,最后讨论的暂定结果是修复BUG的时候,单独切换流水线触发的分支到BUG修复分支上,暂时没有发现其他更好的方式,欢迎大家分享。
流水线实现
-
研发在本地开发完成代码后,push到对应的gitlab,此处为dev分支,push触发Jenkins流水线,执行CI流程,包括单元测试、代码检测、编译打包、推送、并对配置仓库的dev分支进行更新,在这个过程中,会生成image并推送到harbor的demo-dev项目中,并将新生成的image的tag更新到helm的dev分支。整个过程可以增加多步审核,当然,开发环境的部署时研发人员的自主行为,一般不需要做审核操作。
pipeline { agent { node { label 'maven' } } environment { DOCKER_CREDENTIAL_ID = 'harbor-test' //在K8S集群内配置的secrect GITLAB_CREDENTIAL_ID = 'gitlab' //在K8S集群内配置的secrect REGISTRY = 'harbor-test.example.com' DOCKERHUB_NAMESPACE = 'demo' APP_NAME = 'demo' SONAR_CREDENTIAL_ID = 'sonar-token' //在K8S集群内配置的secrect } stages { stage('unit test') { steps { container('maven') { sh 'mvn clean test' } } } stage('sonarqube analysis') { steps { container('maven') { withCredentials([string(credentialsId: "$SONAR_CREDENTIAL_ID", variable: 'SONAR_TOKEN')]) { withSonarQubeEnv('sonar') { sh 'mvn sonar:sonar -Dsonar.login=$SONAR_TOKEN' } } timeout(time: 1, unit: 'HOURS') { waitForQualityGate abortPipeline: true } } } } stage('build & push') { when { branch 'dev' //可以根据多分支配置不通的执行条件,具体情况根据实际需求制定 } steps { container('maven') { script{ env.ctag=sh(script: 'git log --pretty=format:"%H" -n 1', returnStdout: true).trim() env.IMAGE_TAG=sh(script: 'echo ${ctag:0:10}', returnStdout: true).trim() } sh 'mvn clean package -DskipTests' sh 'docker build -f Dockerfile -t $REGISTRY/$DOCKERHUB_NAMESPACE/$APP_NAME:$IMAGE_TAG .' withCredentials([usernamePassword(passwordVariable: 'DOCKER_PASSWORD', usernameVariable: 'DOCKER_USERNAME', credentialsId: "$DOCKER_CREDENTIAL_ID",)]) { sh 'echo "$DOCKER_PASSWORD" | docker login $REGISTRY -u "$DOCKER_USERNAME" --password-stdin' sh 'docker push $REGISTRY/$DOCKERHUB_NAMESPACE/$APP_NAME:$IMAGE_TAG' } } } } stage('update docker tag') { when { branch 'dev' //可以根据多分支配置不通的执行条件,具体情况根据实际需求制定 } environment { BUILD_USER = 'wangjb' BUILD_USER_EMAIL = 'wangjb@jovision.com' YAML_REPO_URL='http://${username}:${password}@gitlab.example.com/cloud-native/helm-demo.git' } steps { withCredentials([usernamePassword(passwordVariable : 'password' ,usernameVariable : 'username' ,credentialsId : "$GITLAB_CREDENTIAL_ID" ,)]) { sh """ git config --global user.name "$BUILD_USER" git config --global user.email "$BUILD_USER_EMAIL" git clone ${YAML_REPO_URL} && cd helm-demo sed -i 's/imagetag:.*/imagetag: "'$IMAGE_TAG'"/' demo/values.yaml git add -A && git commit -m "update tag: ${IMAGE_TAG}" && git push ${YAML_REPO_URL} """ } } } } }
-
ArgoCD检测到helm的dev分支配置与cluster-dev不一致,进行sync操作,从harbor的demo-dev拉取指定的image:tag1。
-
经过dev环境验证后,将helm的dev分支merge到test分支,同时触发流水线将harbor的demo-dev的相应image复制到demo-test,ArgoCD检测配置并进行自动同步,并从demo-test拉取image:tag1。
pipeline {
agent any
environment {
DOCKER_CREDENTIAL_ID = 'harbor-test' //在K8S集群内配置的secrect
DOCKER_CREDENTIAL_ID_PROD = 'harbor-prod' //在K8S集群内配置的secrect
GITHUB_CREDENTIAL_ID = 'gitlab' //在K8S集群内配置的secrect
REGISTRY_TEST = 'http://harbor-test.example.com'
REGISTRY_PROD = 'http://harbor.example.com'
PROJECT_NAME = "demo-test"
REPOSITORY_NAME = 'demo' //The name of the repository. If it contains slash, encode it with URL encoding. e.g. a/b -> a%252Fb
ARTIFACT_FROM = "demo/demo"
}
stages {
stage('镜像同步') {
when {
branch 'prod'
}
steps {
container('base') {
script{
env.ARTIFACT_TAG=sh(script: 'cat demo/values.yaml | grep "imagetag:" | awk -F "\\"" \'{print $2}\'', returnStdout: true).trim()
}
withCredentials([usernamePassword(passwordVariable: 'DOCKER_PASSWORD', usernameVariable: 'DOCKER_USERNAME', credentialsId: "$DOCKER_CREDENTIAL_ID",)]) {
sh '''
echo "COPY ARTIFACT"
result=`curl -X 'POST' -u $DOCKER_USERNAME:$DOCKER_PASSWORD -i \
"$REGISTRY_TEST/api/v2.0/projects/$PROJECT_NAME/repositories/$REPOSITORY_NAME/artifacts?from=$ARTIFACT_FROM:$ARTIFACT_TAG" \
-H 'accept: application/json' | grep "HTTP/1.1 201 Created" | grep -v grep | wc -l`
if [ $result -eq 1 ];then
echo "The artifact copy is successful!"
else
echo "The artifact copy is faild!"
exit 1
fi
'''
}
}
}
}
}
}
- 如果是需要修改helm的其他配置,则可以直接拉取helm-dev、修改后直接推送,之后接merge等操作部署到其他环境即可。
- 多环境(pre-prod、prod等)按同理进行拓展即可
消息通知
Jenkins因为可以使用shell和groovy脚本,因此可以直接调用webhook对接通知接收工具如飞书、钉钉等,此处不做展开。
ArgoCD可以监控应用状态变化,但是它本身却没有能力将变化发送通知,这时候我们就可以使用argocd-notifications
argocd-notifications本身是支持Slack, SMTP, Opsgenie, Telegram 和其他任何使用webhooks的应用。因此我们可以通过使用它的webhook来发送飞书或者钉钉消息。
安装部署
下载并进行修改
首先将配置文件下载到本地:
wget https://raw.githubusercontent.com/argoproj-labs/argocd-notifications/master/manifests/install.yaml
然后对configmap部分进行修改
apiVersion: v1
kind: ConfigMap
metadata:
creationTimestamp: null
name: argocd-notifications-cm
data:
service.webhook.feishu: |
url: https://open.feishu.cn/open-apis/bot/v2/hook/******-******-****** #飞书自定义机器人的webhook
headers:
- name: Content-Type
value: application/json
context: |
argocdUrl: https://www.argocd.com #argocd的web页面,用于消息通知中直接跳转
clusterUrl: http://www.k8scluster.com #K8Sweb地址,用于在消息中直接跳转到集群中的相关页面
template.app-sync-healthy: |
webhook:
feishu:
method: POST
body: |
{
"msg_type": "interactive",
"card": {
"elements": [
{
"tag": "column_set",
"flex_mode": "none",
"background_style": "default",
"columns": [
{
"tag": "column",
"width": "weighted",
"weight": 1,
"vertical_align": "top",
"elements": [
{
"tag": "div",
"text": {
"content": "**⚙ app名称:**\n {{.app.metadata.name}}",
"tag": "lark_md"
}
}
]
},
{
"tag": "column",
"width": "weighted",
"weight": 1,
"vertical_align": "top",
"elements": [
{
"tag": "div",
"text": {
"content": "**⏱ 时间:**\n {{ (call .time.Parse .app.status.operationState.startedAt).Local.Format "2006-01-02 15:04:05(Z07:00)" }}",
"tag": "lark_md"
}
}
]
}
]
},
{
"tag": "column_set",
"flex_mode": "none",
"background_style": "default",
"columns": [
{
"tag": "column",
"width": "weighted",
"weight": 1,
"vertical_align": "top",
"elements": [
{
"tag": "div",
"text": {
"content": "**✔ app同步状态:**\n {{.app.status.operationState.phase}}",
"tag": "lark_md"
}
}
]
},
{
"tag": "column",
"width": "weighted",
"weight": 1,
"vertical_align": "top",
"elements": [
{
"tag": "div",
"text": {
"content": "**💚 app状态:**\n {{.app.status.health.status}}",
"tag": "lark_md"
}
}
]
}
]
},
{
"tag": "column_set",
"flex_mode": "none",
"background_style": "default",
"columns": [
{
"tag": "column",
"width": "weighted",
"weight": 1,
"vertical_align": "top",
"elements": [
{
"tag": "div",
"text": {
"content": "**☸ 所属集群:**\n {{ .app.spec.destination.name}}",
"tag": "lark_md"
}
}
]
},
{
"tag": "column",
"width": "weighted",
"weight": 1,
"vertical_align": "top",
"elements": [
{
"tag": "div",
"text": {
"content": "**⚪ 集群NS:**\n {{ .app.spec.destination.namespace}}",
"tag": "lark_md"
}
}
]
}
]
},
{
"tag": "action",
"actions": [
{
"tag": "button",
"text": {
"tag": "plain_text",
"content": "跳转ArgoCD"
},
"type": "primary",
"multi_url": {
"url": "{{.context.argocdUrl}}/applications/{{.app.metadata.name}}?operation=true",
"pc_url": "",
"android_url": "",
"ios_url": ""
}
},
{
"tag": "button",
"text": {
"tag": "plain_text",
"content": "跳转Deployment"
},
"type": "default",
"multi_url": {
"url": "{{.context.clusterUrl}}/awbaby/clusters/{{.app.spec.destination.name}}/projects/{{.app.metadata.name}}/deployments",
"pc_url": "",
"android_url": "",
"ios_url": ""
}
}
]
},
{
"tag": "hr"
},
{
"tag": "div",
"fields": [
{
"is_short": true,
"text": {
"tag": "lark_md",
"content": "**Gitlab CommitID:**\n{{ .app.status.operationState.operation.sync.revision}}\n"
}
}
]
}
],
"header": {
"template": "green",
"title": {
"content": "ArgoCD同步通知",
"tag": "plain_text"
}
}
}
}
template.app-sync-unhealthy: |
webhook:
feishu:
method: POST
body: |
{
"msg_type": "interactive",
"card": {
"elements": [
{
"tag": "column_set",
"flex_mode": "none",
"background_style": "default",
"columns": [
{
"tag": "column",
"width": "weighted",
"weight": 1,
"vertical_align": "top",
"elements": [
{
"tag": "div",
"text": {
"content": "**⚙ app名称:**\n {{.app.metadata.name}}",
"tag": "lark_md"
}
}
]
},
{
"tag": "column",
"width": "weighted",
"weight": 1,
"vertical_align": "top",
"elements": [
{
"tag": "div",
"text": {
"content": "**⏱ 时间:**\n {{ (call .time.Parse .app.status.operationState.startedAt).Local.Format "2006-01-02 15:04:05(Z07:00)" }}",
"tag": "lark_md"
}
}
]
}
]
},
{
"tag": "column_set",
"flex_mode": "none",
"background_style": "default",
"columns": [
{
"tag": "column",
"width": "weighted",
"weight": 1,
"vertical_align": "top",
"elements": [
{
"tag": "div",
"text": {
"content": "**✔ app同步状态:**\n {{.app.status.operationState.phase}}",
"tag": "lark_md"
}
}
]
},
{
"tag": "column",
"width": "weighted",
"weight": 1,
"vertical_align": "top",
"elements": [
{
"tag": "div",
"text": {
"content": "**💔 app状态:**\n {{.app.status.health.status}}",
"tag": "lark_md"
}
}
]
}
]
},
{
"tag": "column_set",
"flex_mode": "none",
"background_style": "default",
"columns": [
{
"tag": "column",
"width": "weighted",
"weight": 1,
"vertical_align": "top",
"elements": [
{
"tag": "div",
"text": {
"content": "**☸ 所属集群:**\n {{ .app.spec.destination.name}}",
"tag": "lark_md"
}
}
]
},
{
"tag": "column",
"width": "weighted",
"weight": 1,
"vertical_align": "top",
"elements": [
{
"tag": "div",
"text": {
"content": "**⚪ 集群NS:**\n {{ .app.spec.destination.namespace}}",
"tag": "lark_md"
}
}
]
}
]
},
{
"tag": "action",
"actions": [
{
"tag": "button",
"text": {
"tag": "plain_text",
"content": "跳转ArgoCD"
},
"type": "primary",
"multi_url": {
"url": "{{.context.argocdUrl}}/applications/{{.app.metadata.name}}?operation=true",
"pc_url": "",
"android_url": "",
"ios_url": ""
}
},
{
"tag": "button",
"text": {
"tag": "plain_text",
"content": "跳转Deployment"
},
"type": "default",
"multi_url": {
"url": "{{.context.clusterUrl}}/awbaby/clusters/{{.app.spec.destination.name}}/projects/{{.app.metadata.name}}/deployments",
"pc_url": "",
"android_url": "",
"ios_url": ""
}
}
]
},
{
"tag": "hr"
},
{
"tag": "div",
"fields": [
{
"is_short": true,
"text": {
"tag": "lark_md",
"content": "**Gitlab CommitID:**\n{{ .app.status.operationState.operation.sync.revision}}\n"
}
}
]
}
],
"header": {
"template": "red",
"title": {
"content": "ArgoCD同步通知",
"tag": "plain_text"
}
}
}
}
template.app-sync-err-unhealthy: |
webhook:
feishu:
method: POST
body: |
{
...
}
template.app-sync-running: |
webhook:
feishu:
method: POST
body: |
{
...
}
template.app-sync-change-succ: |
webhook:
feishu:
method: POST
body: |
{
...
}
trigger.on-deployed: |
- description: Application is synced and healthy. Triggered once per commit.
oncePer: app.status.operationState.operation.sync.revision
send: [app-sync-healthy] # template names
# trigger condition
when: app.status.operationState.phase in ['Succeeded'] and app.status.health.status == 'Healthy'
trigger.on-health-degraded: |
- description: Application has degraded
send: [app-sync-unhealthy]
when: app.status.operationState.phase in ['Succeeded'] and app.status.health.status in ['Degraded','Unknown']
trigger.on-sync-failed: |
- description: Application syncing has failed
send: [app-sync-err-unhealthy] # template names
when: app.status.operationState.phase in ['Error','Failed']
trigger.on-sync-running: |
- description: Application is being synced
send: [app-sync-running] # template names
when: app.status.operationState.phase in ['Running']
trigger.on-sync-status-unknown: |
- description: Application status is 'Unknown,Degraded' and Application syncing has failed
send: [app-sync-err-unhealthy] # template names
when: app.status.operationState.phase in ['Error','Failed'] and app.status.sync.status in ['Degraded','Unknown']
trigger.on-sync-succeeded: |
- description: Application syncing has succeeded
send: [app-sync-change-succ] # template names
when: app.status.operationState.phase in ['Succeeded']
subscriptions: |
- recipients: [feishu] # 可能有bug,正常应该是webhook:feishu
triggers: [on-sync-running, on-deployed, on-sync-failed, on-sync-succeeded,on-health-degraded]
-
template.app-sync-healthy、app-sync-err-unhealthy、app-sync-running等等都是可以根据自己的需要自行制定消息模板,我这个是基于飞书卡片通知的,具体模板制定请参考飞书官方,同样,也可以制定钉钉的消息模板,方式类似,具体可以参考钉钉官方文档
-
trigger.on-deployed、trigger.on-health-degraded、trigger.on-sync-failed这些是触发器,即在满足什么条件的情况下,会触发使用哪个模板发送消息
subscriptions是指定启用哪些触发器,还有更多详细功能,建议参考官方文档进行合适的定制开发。
执行安装
使用kubectl部署到argocd的namespace中
kubectl apply -f install.yaml -n argocd
当pod就绪后,就可以开始测试了,当然在执行项目实际测试之前,可以先手动触发测试
kubectl exec -it argocd-notifications-controller-5685f9ccbf-n7plj -n argocd -- /app/argocd-notifications template notify app-sync-err-unhealthy demo --recipient feishu
小技巧分享
最后推荐一个网站:https://unicode.org/emoji/charts/full-emoji-list.html
在shell中有些情况需要使用表情符的时候,可以参考这个代码,我飞书通知中使用的某些表情符号就是在这找的。
使用方法,比如说要显示🧡,在vim中使用大写锁定键,然后按ctrl+v,在^状态下,分别输入U1F9E1,不需要写输入那个“+”,然后ECS退出编辑或者回车都可以显示出符号,但是这个符号在shell中是没有颜色的,在跟随其他的消息,如飞书通知等输出后,就是我们想要的符号了。