通过 karmada + istio + argo-rollout 进行多集群编排(2023-11-12)
后端发布方案:
四个集群,每个集群都部署 Rollout、Services
mkdir -p ~/helloworld-rollout-yml
cat > helloworld-stable-rollout.yml << 'EOF'
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
name: helloworld-stable
namespace: helloworld-rollout
spec:
replicas: 4 # 副本数
strategy:
canary:
steps:
- setWeight: 20
- pause: {} # 人工卡点
- setWeight: 40
- pause: {duration: 10}
- setWeight: 60
- pause: {duration: 10}
- setWeight: 80
- pause: {duration: 10}
- setWeight: 100
- pause: {} # 人工卡点
revisionHistoryLimit: 2
selector:
matchLabels:
app: helloworld
template:
metadata:
labels:
app: helloworld
spec:
containers:
- name: helloworld
image: ccr.ccs.tencentyun.com/huanghuanhui/helloworld:stable
imagePullPolicy: IfNotPresent
ports:
- name: http
containerPort: 80
protocol: TCP
EOF
cat > helloworld-stable-svc.yml << 'EOF'
apiVersion: v1
kind: Service
metadata:
name: helloworld
namespace: helloworld-rollout
labels:
app: helloworld
service: helloworld
spec:
ports:
- port: 80
name: http
selector:
app: helloworld
EOF
cat > helloworld-stable-policy.yml << 'EOF'
apiVersion: policy.karmada.io/v1alpha1
kind: PropagationPolicy
metadata:
name: helloworld-stable-propagation
namespace: helloworld-rollout
spec:
resourceSelectors:
- apiVersion: argoproj.io/v1alpha1 # 修正为 Argo Rollout 的 apiVersion
kind: Rollout
name: helloworld-stable
- apiVersion: v1
kind: Service
name: helloworld
placement:
clusterAffinity:
clusterNames:
- k8s-master-beijing
- k8s-master-shanghai
- k8s-master-guangzhou
- k8s-master-shenzhen
replicaScheduling:
replicaDivisionPreference: Weighted
replicaSchedulingType: Divided
weightPreference:
staticWeightList:
- targetCluster:
clusterNames:
- k8s-master-beijing
weight: 1
- targetCluster:
clusterNames:
- k8s-master-shanghai
weight: 1
- targetCluster:
clusterNames:
- k8s-master-guangzhou
weight: 1
- targetCluster:
clusterNames:
- k8s-master-shenzhen
weight: 1
EOF
共部署4个副本,每个集群分配一个副本(这里以四个集群为例!!!)
前端发布方案(无金丝雀):
四个集群,每个集群都部署 Rollout、Services、Istio VirtualService 和 Istio Gateway
mkdir -p ~/helloworld-rollout-yml
cat > helloworld-stable-rollout.yml << 'EOF'
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
name: helloworld-stable
namespace: helloworld-rollout
spec:
replicas: 4
strategy:
canary:
steps:
- setWeight: 20
- pause: {} # 人工卡点
- setWeight: 40
- pause: {duration: 10}
- setWeight: 60
- pause: {duration: 10}
- setWeight: 80
- pause: {duration: 10}
- setWeight: 100
- pause: {} # 人工卡点
revisionHistoryLimit: 2
selector:
matchLabels:
app: helloworld
template:
metadata:
labels:
app: helloworld
spec:
containers:
- name: helloworld
image: ccr.ccs.tencentyun.com/huanghuanhui/helloworld:stable
imagePullPolicy: IfNotPresent
ports:
- name: http
containerPort: 80
protocol: TCP
EOF
cat > helloworld-stable-svc.yml << 'EOF'
apiVersion: v1
kind: Service
metadata:
name: helloworld
namespace: helloworld-rollout
labels:
app: helloworld
service: helloworld
spec:
ports:
- port: 80
name: http
selector:
app: helloworld
EOF
cat > helloworld-stable-vsvc.yml << 'EOF'
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: helloworld-vsvc
namespace: helloworld-rollout
spec:
gateways:
- helloworld-gateway
hosts:
- "helloworld.huanghuanhui.cloud"
http:
- name: primary
route:
- destination:
host: prd-vue-svc-stable
weight: 100
- destination:
host: prd-vue-svc-canary
weight: 0
EOF
cat > helloworld-stable-gateway.yml << 'EOF'
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: helloworld-gateway
namespace: helloworld-rollout
spec:
selector:
istio: ingressgateway # 默认创建的 istio ingressgateway pod 有这个 Label
servers:
- port:
number: 443
name: https
protocol: HTTPS
hosts:
- "helloworld.huanghuanhui.cloud" # 匹配所有 host
tls:
mode: SIMPLE
credentialName: helloworld-rollout-tls-secret
EOF
# 所有的istio的证书都放在(istio-system)命名空间下
kubectl create secret -n istio-system \
tls prd-vue-tls-secret \
--key=/root/ssl/huanghuanhui.cloud_nginx/huanghuanhui.cloud.key \
--cert=/root/ssl/huanghuanhui.cloud_nginx/huanghuanhui.cloud_bundle.crt
cat > helloworld-stable-policy.yml << 'EOF'
apiVersion: policy.karmada.io/v1alpha1
kind: PropagationPolicy
metadata:
name: helloworld-stable-propagation
namespace: helloworld-rollout
spec:
resourceSelectors:
- apiVersion: argoproj.io/v1alpha1 # 修正为 Argo Rollout 的 apiVersion
kind: Rollout
name: helloworld-stable
- apiVersion: v1
kind: Service
name: helloworld
placement:
clusterAffinity:
clusterNames:
- k8s-master-beijing
- k8s-master-shanghai
- k8s-master-guangzhou
- k8s-master-shenzhen
replicaScheduling:
replicaDivisionPreference: Weighted
replicaSchedulingType: Divided
weightPreference:
staticWeightList:
- targetCluster:
clusterNames:
- k8s-master-beijing
weight: 1
- targetCluster:
clusterNames:
- k8s-master-shanghai
weight: 1
- targetCluster:
clusterNames:
- k8s-master-guangzhou
weight: 1
- targetCluster:
clusterNames:
- k8s-master-shenzhen
weight: 1
EOF
共部署4个副本,每个集群分配一个副本(这里以四个集群为例!!!)
前端发布方案(金丝雀):
四个集群,每个集群都部署 Rollout、Services、Istio VirtualService 和 Istio Gateway
mkdir -p ~/helloworld-rollout-yml
cat > helloworld-rollout.yml << 'EOF'
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
name: helloworld
namespace: helloworld-rollout
spec:
replicas: 4
strategy:
canary:
canaryService: helloworld-svc-canary # 关联 canary Service
stableService: helloworld-svc-stable # 关联 stable Service
trafficRouting:
managedRoutes:
- name: "header-route-1"
istio:
virtualServices:
- name: helloworld-vsvc # 关联的 Istio virtualService
routes:
- primary
steps:
- setHeaderRoute:
name: "header-route-1"
match:
- headerName: "X-canary"
headerValue:
exact: "test-user"
- pause: {duration: 10}
- setCanaryScale:
weight: 20
- pause: {} # 人工卡点(当有新版本上线的时候,给canary版本20%流量,并且暂停新版本继续更新,当测试人员完成测试可以继续更新,从而达到前端可以一直触发构建并且直接上到金丝雀上)
- setCanaryScale:
weight: 40
- pause: {duration: 10}
- setCanaryScale:
weight: 60
- pause: {duration: 10}
- setCanaryScale:
weight: 80
- pause: {duration: 10}
revisionHistoryLimit: 5
selector:
matchLabels:
app: helloworld
template:
metadata:
labels:
app: helloworld
spec:
containers:
- name: helloworld
image: ccr.ccs.tencentyun.com/huanghuanhui/helloworld:canary
imagePullPolicy: IfNotPresent
ports:
- name: http
containerPort: 80
protocol: TCP
EOF
cat > helloworld-rollout-svc.yml << 'EOF'
apiVersion: v1
kind: Service
metadata:
name: helloworld-svc-stable
namespace: helloworld-rollout
labels:
app: helloworld
spec:
type: ClusterIP
ports:
- port: 80
targetPort: http
protocol: TCP
name: http
selector:
app: helloworld
---
apiVersion: v1
kind: Service
metadata:
name: helloworld-svc-canary
namespace: helloworld-rollout
labels:
app: helloworld
spec:
type: ClusterIP
ports:
- port: 80
targetPort: http
protocol: TCP
name: http
selector:
app: helloworld
EOF
cat > helloworld-vsvc.yml << 'EOF'
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: helloworld-vsvc
namespace: helloworld-rollout
spec:
gateways:
- helloworld-gateway
hosts:
- "helloworld.huanghuanhui.cloud"
http:
- name: primary
route:
- destination:
host: helloworld-svc-stable
weight: 100
- destination:
host: helloworld-svc-canary
weight: 0
EOF
cat > helloworld-gateway.yml << 'EOF'
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: helloworld-gateway
namespace: helloworld-rollout
spec:
selector:
istio: ingressgateway # 默认创建的 istio ingressgateway pod 有这个 Label
servers:
- port:
number: 443
name: https
protocol: HTTPS
hosts:
- "helloworld.huanghuanhui.cloud" # 匹配所有 host
tls:
mode: SIMPLE
credentialName: helloworld-rollout-tls-secret
EOF
# 所有的istio的证书都放在(istio-system)命名空间下
kubectl create secret -n istio-system \
tls prd-vue-tls-secret \
--key=/root/ssl/huanghuanhui.cloud_nginx/huanghuanhui.cloud.key \
--cert=/root/ssl/huanghuanhui.cloud_nginx/huanghuanhui.cloud_bundle.crt
cat > helloworld-stable-policy.yml << 'EOF'
apiVersion: policy.karmada.io/v1alpha1
kind: PropagationPolicy
metadata:
name: helloworld-stable-propagation
namespace: helloworld-rollout
spec:
resourceSelectors:
- apiVersion: argoproj.io/v1alpha1 # 修正为 Argo Rollout 的 apiVersion
kind: Rollout
name: helloworld-stable
- apiVersion: v1
kind: Service
name: helloworld
placement:
clusterAffinity:
clusterNames:
- k8s-master-beijing
- k8s-master-shanghai
- k8s-master-guangzhou
- k8s-master-shenzhen
replicaScheduling:
replicaDivisionPreference: Weighted
replicaSchedulingType: Divided
weightPreference:
staticWeightList:
- targetCluster:
clusterNames:
- k8s-master-beijing
weight: 1
- targetCluster:
clusterNames:
- k8s-master-shanghai
weight: 1
- targetCluster:
clusterNames:
- k8s-master-guangzhou
weight: 1
- targetCluster:
clusterNames:
- k8s-master-shenzhen
weight: 1
EOF
共部署4个副本,每个集群分配一个副本(这里以四个集群为例!!!)
===
1、ruoyi-gateway(部署 Rollout、Services、Istio VirtualService 和 Istio Gateway)
2、ruoyi-auth(部署 Rollout)
3、ruoyi-system(部署 Rollout)
4、ruoyi-vue(部署 Rollout、Services、Istio VirtualService 和 Istio Gateway)
正常的流量走vue的stable版本,连接stable的gateway
带请求头的流量走vue的canary版本,连接canary版本的gateway
访问前端canary版本,后端也走canary版本;访问前端stable版本,后端也走stable版本;
问题1:ruoyi-auth、ruoyi-system怎样区分流量???