aws appmesh servicemesh 通过virtual gateway公开流量

zhaojiew10

已于 2024-05-28 20:50:08 修改

阅读量380

点赞数 1

分类专栏： AWS 文章标签： aws docker 云计算

于 2022-10-16 01:43:00 首次发布

本文链接：https://blog.csdn.net/sinat_41567654/article/details/127342935

版权

AWS 专栏收录该内容

72 篇文章 4 订阅

订阅专栏

该product增加了virtual gateway 的demo，还可以按需开启xray

资源仓库

git clone git@github.com:aws-containers/eks-app-mesh-polyglot-demo.git

项目的架构如下：
在这里插入图片描述
frontend-node 和 product detail 部署在managed node上，product catalog 部署在fargate上，箭头为调用链

部署示例业务

按照上一篇文档部署集群可观察性，https://blog.csdn.net/sinat_41567654/article/details/127342629

按照说明将image打包到ecr中，典型的dockerfile如下

FROM node:14
WORKDIR /usr/src/app
COPY package.json package-lock.json ./
RUN npm ci --only=production
COPY . .
EXPOSE 9000
CMD [ "node", "server.js" ]

打包镜像到ecr

aws ecr get-login-password --region $AWS_REGION | docker login --username AWS --password-stdin $ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com
PROJECT_NAME=eks-app-mesh-demo
export APP_VERSION=1.0
for app in catalog_detail product_catalog frontend_node; do
  aws ecr describe-repositories --repository-name $PROJECT_NAME/$app >/dev/null 2>&1 || \
  aws ecr create-repository --repository-name $PROJECT_NAME/$app >/dev/null
  TARGET=$ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com/$PROJECT_NAME/$app:$APP_VERSION
  docker build -t $TARGET apps/$app
  docker push $TARGET
done

确认如下image
在这里插入图片描述
部署资源，需要修改image名称，或者设置环境变量

注意：创建fargate需要eks集群的fargate配置

envsubst < ./deployment/base_app.yaml | kubectl apply -f -

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: frontend-node
  namespace: prodcatalog-ns
spec:
  replicas: 1
  selector:
    matchLabels:
      app: frontend-node
  template:
    metadata:
      annotations:
        prometheus.io/scrape: 'true'
        prometheus.io/path: '/stats/prometheus'
      labels:
        app: frontend-node
    spec:
      serviceAccountName: prodcatalog-envoy-proxies
      containers:
        - name: frontend-node
          image: "${ACCOUNT_ID}.dkr.ecr.${AWS_REGION}.amazonaws.com/eks-app-mesh-demo/frontend_node:${APP_VERSION}"
          imagePullPolicy: Always
          livenessProbe:
            httpGet:
              path: /ping
              port: 9000
            initialDelaySeconds: 0
            periodSeconds: 10
            timeoutSeconds: 1
            failureThreshold: 3
          readinessProbe:
            httpGet:
              path: /ping
              port: 9000
            successThreshold: 3
          env:
            - name: BASE_URL
              value: "http://prodcatalog.prodcatalog-ns.svc.cluster.local:5000/products/"
          ports:
            - containerPort: 9000
---
apiVersion: v1
kind: Service
metadata:
  #annotations:
  # This annotation is only required if you are creating an internal facing ELB. Remove this annotation to create public facing ELB.
  #service.beta.kubernetes.io/aws-load-balancer-internal: "true"
  name: frontend-node
  namespace: prodcatalog-ns
  labels:
    app: frontend-node
spec:
  ports:
    - name: "http"
      port: 9000
      targetPort: 9000
  selector:
    app: frontend-node
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: proddetail
  namespace: prodcatalog-ns
spec:
  replicas: 1
  selector:
    matchLabels:
      app: proddetail
  template:
    metadata:
      labels:
        app: proddetail
    spec:
      serviceAccountName: prodcatalog-envoy-proxies
      containers:
        - name: proddetail
          image: "${ACCOUNT_ID}.dkr.ecr.${AWS_REGION}.amazonaws.com/eks-app-mesh-demo/catalog_detail:${APP_VERSION}"
          imagePullPolicy: Always
          livenessProbe:
            httpGet:
              path: /ping
              port: 3000
            initialDelaySeconds: 0
            periodSeconds: 10
            timeoutSeconds: 1
            failureThreshold: 3
          readinessProbe:
            httpGet:
              path: /ping
              port: 3000
            successThreshold: 3
          ports:
            - containerPort: 3000
---
apiVersion: v1
kind: Service
metadata:
  #annotations:
  # This annotation is only required if you are creating an internal facing ELB. Remove this annotation to create public facing ELB.
  #service.beta.kubernetes.io/aws-load-balancer-internal: "true"
  name: proddetail
  namespace: prodcatalog-ns
  labels:
    app: proddetail
spec:
  ports:
    - name: "http"
      port: 3000
      targetPort: 3000
  selector:
    app: proddetail
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: prodcatalog
  namespace: prodcatalog-ns
spec:
  replicas: 1
  selector:
    matchLabels:
      app: prodcatalog
  template:
    metadata:
      labels:
        app: prodcatalog
    spec:
      serviceAccountName: prodcatalog-envoy-proxies
      containers:
        - name: prodcatalog
          image: "${ACCOUNT_ID}.dkr.ecr.${AWS_REGION}.amazonaws.com/eks-app-mesh-demo/product_catalog:${APP_VERSION}"
          imagePullPolicy: Always
          env:
            - name: AGG_APP_URL
              value: "http://proddetail.prodcatalog-ns.svc.cluster.local:3000/catalogDetail"
          livenessProbe:
            httpGet:
              path: /products/ping
              port: 5000
            initialDelaySeconds: 0
            periodSeconds: 10
            timeoutSeconds: 1
            failureThreshold: 3
          readinessProbe:
            httpGet:
              path: /products/ping
              port: 5000
            successThreshold: 3
          ports:
            - containerPort: 5000
---
apiVersion: v1
kind: Service
metadata:
  name: prodcatalog
  namespace: prodcatalog-ns
  labels:
    app: prodcatalog
spec:
  ports:
    - name: "http"
      port: 5000
      targetPort: 5000
  selector:
    app: prodcatalog
---

此时，并没有将sidecar注入到pod中appmesh controller会监控k8s对象，当在集群中创建相关资源时会创建对应的aws应用

pod需要注入sidecar代理才能加入网格，应用网格使用 Envoy sidecar 容器作为到主服务的所有入口和出口流量
在这里插入图片描述
将资源加入网格之后的架构

测试连接，在集群内使用front node访问

root@frontend-node-dcc4bf8b9-pwmdz:/usr/src/app# curl http://prodcatalog.prodcatalog-ns.svc.cluster.local:5000/products/ 
{
    "products": {
        "1": "Table",
        "2": "Chair"
    },
    "details": {
        "version": "1",
        "vendors": [
            "ABC.com"
        ]
    }
}

使用fargate访问后端

root@prodcatalog-65cfb85f94-k44b4:/app# curl http://proddetail.prodcatalog-ns.svc.cluster.local:3000/catalogDetail
{"version":"1","vendors":["ABC.com"]}

部署appmesh

参考 https://blog.csdn.net/sinat_41567654/article/details/127330432 进行appmesh controller 和CRD资源的部署

将业务容器和服务加入网格之后的调用链应当如下
在这里插入图片描述
创建mesh

当注入prodcatalog-ns命名空间后，之前启动的pod都需要rollout restart进行更新，此外pod需要appmesh的serviceaccount权限

---
apiVersion: v1
kind: Namespace
metadata:
  name: prodcatalog-ns
  labels:
    mesh: prodcatalog-mesh
    gateway: ingress-gw
    appmesh.k8s.aws/sidecarInjectorWebhook: enabled
---
apiVersion: appmesh.k8s.aws/v1beta2
kind: Mesh
metadata:
  name: prodcatalog-mesh
spec:
  namespaceSelector:
    matchLabels:
      mesh: prodcatalog-mesh
---

创建mesh内部资源

---
apiVersion: appmesh.k8s.aws/v1beta2
kind: VirtualNode
metadata:
  name: prodcatalog
  namespace: prodcatalog-ns
spec:
  podSelector:
    matchLabels:
      app: prodcatalog
  listeners:
    - portMapping:
        port: 5000
        protocol: http
      healthCheck:
        protocol: http
        path: '/products/ping'
        healthyThreshold: 2
        unhealthyThreshold: 2
        timeoutMillis: 2000
        intervalMillis: 5000
  backends:
    - virtualService:
        virtualServiceRef:
          name: proddetail
  serviceDiscovery:
    dns:
      hostname: prodcatalog.prodcatalog-ns.svc.cluster.local
  logging:
    accessLog:
      file:
        path: /dev/stdout
---
apiVersion: appmesh.k8s.aws/v1beta2
kind: VirtualService
metadata:
  name: prodcatalog
  namespace: prodcatalog-ns
spec:
  awsName: prodcatalog.prodcatalog-ns.svc.cluster.local
  provider:
    virtualRouter:
      virtualRouterRef:
        name: prodcatalog-router
---
apiVersion: appmesh.k8s.aws/v1beta2
kind: VirtualService
metadata:
  name: proddetail
  namespace: prodcatalog-ns
spec:
  awsName: proddetail.prodcatalog-ns.svc.cluster.local
  provider:
    virtualRouter:
      virtualRouterRef:
        name: proddetail-router
---
apiVersion: appmesh.k8s.aws/v1beta2
kind: VirtualRouter
metadata:
  name: proddetail-router
  namespace: prodcatalog-ns
spec:
  listeners:
    - portMapping:
        port: 3000
        protocol: http
  routes:
    - name: proddetail-route
      httpRoute:
        match:
          prefix: /
        action:
          weightedTargets:
            - virtualNodeRef:
                name: proddetail-v1
              weight: 100
---
apiVersion: appmesh.k8s.aws/v1beta2
kind: VirtualRouter
metadata:
  name: prodcatalog-router
  namespace: prodcatalog-ns
spec:
  listeners:
    - portMapping:
        port: 5000
        protocol: http
  routes:
    - name: prodcatalog-route
      httpRoute:
        match:
          prefix: /
        action:
          weightedTargets:
            - virtualNodeRef:
                name: prodcatalog
              weight: 100
---
apiVersion: appmesh.k8s.aws/v1beta2
kind: VirtualNode
metadata:
  name: proddetail-v1
  namespace: prodcatalog-ns
spec:
  podSelector:
    matchLabels:
      app: proddetail
  listeners:
    - portMapping:
        port: 3000
        protocol: http
      healthCheck:
        protocol: http
        path: '/ping'
        healthyThreshold: 2
        unhealthyThreshold: 2
        timeoutMillis: 2000
        intervalMillis: 5000
  serviceDiscovery:
    dns:
      hostname: proddetail.prodcatalog-ns.svc.cluster.local
  logging:
    accessLog:
      file:
        path: /dev/stdout
---
apiVersion: appmesh.k8s.aws/v1beta2
kind: VirtualNode
metadata:
  name: frontend-node
  namespace: prodcatalog-ns
spec:
  podSelector:
    matchLabels:
      app: frontend-node
  listeners:
    - portMapping:
        port: 9000
        protocol: http
  backends:
    - virtualService:
        virtualServiceRef:
          name: prodcatalog
    - virtualService:
        virtualServiceRef:
          name: proddetail
  serviceDiscovery:
    dns:
      hostname: frontend-node.prodcatalog-ns.svc.cluster.local
  logging:
    accessLog:
      file:
        path: /dev/stdout
---
apiVersion: appmesh.k8s.aws/v1beta2
kind: VirtualService
metadata:
  name: frontend-node
  namespace: prodcatalog-ns
spec:
  awsName: frontend-node.prodcatalog-ns.svc.cluster.local
  provider:
    virtualNode:
      virtualNodeRef:
        name: frontend-node
---

成功注入
在这里插入图片描述
控制台能看到创建的响应mesh资源

front end 向 catalog发请求，可以看到对后端服务的请求是通过 envoy 代理发出的

Expire in 3 ms for 1 (transfer 0x5632268a50f0)
Expire in 3 ms for 1 (transfer 0x5632268a50f0)
Expire in 4 ms for 1 (transfer 0x5632268a50f0)
  Trying 10.100.253.70...
TCP_NODELAY set
Expire in 200 ms for 4 (transfer 0x5632268a50f0)
Connected to prodcatalog.prodcatalog-ns.svc.cluster.local (10.100.253.70) port 5000 (#0)
GET /products/ HTTP/1.1
Host: prodcatalog.prodcatalog-ns.svc.cluster.local:5000
User-Agent: curl/7.64.0
Accept: */*

HTTP/1.1 200 OK
content-type: application/json
content-length: 124
x-amzn-trace-id: Root=1-634ae7ff-ea71a73215b80836af52575e
access-control-allow-origin: *
server: envoy
date: Sat, 15 Oct 2022 17:03:59 GMT
x-envoy-upstream-service-time: 18


  "products": {},
  "details": {
      "version": "1",
      "vendors": [
          "ABC.com"
      ]
  }

Connection #0 to host prodcatalog.prodcatalog-ns.svc.cluster.local left intact

向mesh外部公开流量

virtualgateway允许mesh外部的资源与mesh内通信

外部资源必须能够将 DNS 名称解析为分配给运行 Envoy 的服务或实例的 IP 地址

virtualgateway传入的流量是gatewayroute指定的
在这里插入图片描述
调用链

部署虚拟网关

可见VirtualGateway本身就是一个没有app容器的envoy代理容器，可以指定后端为其他虚拟服务，但是自身是通过loadblancer暴露的

由于中国区的问题，这里需要修改lb的监听端口

---
apiVersion: appmesh.k8s.aws/v1beta2
kind: VirtualGateway
metadata:
  name: ingress-gw
  namespace: prodcatalog-ns
spec:
  namespaceSelector:
    matchLabels:
      gateway: ingress-gw
  podSelector:
    matchLabels:
      app: ingress-gw
  listeners:
    - portMapping:
        port: 8088
        protocol: http
  logging:
    accessLog:
      file:
        path: /dev/stdout
---
apiVersion: appmesh.k8s.aws/v1beta2
kind: GatewayRoute
metadata:
  name: gateway-route-frontend
  namespace: prodcatalog-ns
spec:
  httpRoute:
    match:
      prefix: "/"
    action:
      target:
        virtualService:
          virtualServiceRef:
            name: frontend-node
---
apiVersion: v1
kind: Service
metadata:
  name: ingress-gw
  namespace: prodcatalog-ns
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-type: "nlb"
spec:
  type: LoadBalancer
  ports:
    - port: 8088
      targetPort: 8088
      name: http
  selector:
    app: ingress-gw
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: ingress-gw
  namespace: prodcatalog-ns
spec:
  replicas: 1
  selector:
    matchLabels:
      app: ingress-gw
  template:
    metadata:
      labels:
        app: ingress-gw
    spec:
      serviceAccountName: prodcatalog-envoy-proxies
      securityContext:
        fsGroup: 65534
      containers:
        - name: envoy
          image: public.ecr.aws/appmesh/aws-appmesh-envoy:v1.23.1.0-prod
          ports:
            - containerPort: 8088
---

创建资源和属性如下
在这里插入图片描述
默认通过intree创建了nlb

尝试访问

curl命令，说明还是通过envoy转发的

$ curl -v http://a75a3e6d266c4428ca2e64cd0fed773b-16dd259f11b9fb85.elb.cn-north-1.amazonaws.com.cn:8088/
*   Trying 52.80.63.251:8088...
* Connected to a75a3e6d266c4428ca2e64cd0fed773b-16dd259f11b9fb85.elb.cn-north-1.amazonaws.com.cn (52.80.63.251) port 8088 (#0)
> GET / HTTP/1.1
> Host: a75a3e6d266c4428ca2e64cd0fed773b-16dd259f11b9fb85.elb.cn-north-1.amazonaws.com.cn:8088
> User-Agent: curl/7.79.1
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< x-powered-by: Express
< content-type: text/html; charset=utf-8
< content-length: 1195
< etag: W/"4ab-ju0cYuWnpkHio52kIUHS0XrmIdU"
< date: Sat, 15 Oct 2022 17:27:06 GMT
< x-envoy-upstream-service-time: 35
< server: envoy

发布新的defatil版本，方便测试将权重调整为55开

---
apiVersion: appmesh.k8s.aws/v1beta2
kind: VirtualNode
metadata:
  name: proddetail-v2
  namespace: prodcatalog-ns
spec:
  podSelector:
    matchLabels:
      app: proddetail2
  listeners:
    - portMapping:
        port: 3000
        protocol: http
      healthCheck:
        protocol: http
        path: '/ping'
        healthyThreshold: 2
        unhealthyThreshold: 2
        timeoutMillis: 2000
        intervalMillis: 5000
  serviceDiscovery:
    dns:
      hostname: proddetail2.prodcatalog-ns.svc.cluster.local
---
apiVersion: appmesh.k8s.aws/v1beta2
kind: VirtualRouter
metadata:
  name: proddetail-router
  namespace: prodcatalog-ns
spec:
  listeners:
    - portMapping:
        port: 3000
        protocol: http
  routes:
    - name: proddetail-route
      httpRoute:
        match:
          prefix: /
        action:
          weightedTargets:
            - virtualNodeRef:
                name: proddetail-v1
              weight: 50
            - virtualNodeRef:
                name: proddetail-v2
              weight: 50
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: proddetail2
  namespace: prodcatalog-ns
spec:
  replicas: 1
  selector:
    matchLabels:
      app: proddetail2
  template:
    metadata:
      labels:
        app: proddetail2
    spec:
      serviceAccountName: prodcatalog-envoy-proxies
      containers:
        - name: proddetail
          image: "${ACCOUNT_ID}.dkr.ecr.${AWS_REGION}.amazonaws.com/eks-app-mesh-demo/catalog_detail:${APP_VERSION_2}"
          imagePullPolicy: Always
          livenessProbe:
            httpGet:
              path: /ping
              port: 3000
            initialDelaySeconds: 0
            periodSeconds: 10
            timeoutSeconds: 1
            failureThreshold: 3
          readinessProbe:
            httpGet:
              path: /ping
              port: 3000
            successThreshold: 3
          ports:
            - containerPort: 3000
---
apiVersion: v1
kind: Service
metadata:
  name: proddetail2
  namespace: prodcatalog-ns
  labels:
    app: proddetail2
spec:
  ports:
    - name: "http"
      port: 3000
      targetPort: 3000
  selector:
    app: proddetail2
---

在这里插入图片描述
测试路由生效

root@prodcatalog-54d9dc6d9b-ctjkw:/app# for i in $(seq 1 10);do sleep 1;echo $(curl -s http://proddetail.prodcatalog-ns.svc.cluster.local:3000/catalogDetail); done
{"version":"2","vendors":["ABC.com","XYZ.com"]}
{"version":"1","vendors":["ABC.com"]}
{"version":"1","vendors":["ABC.com"]}
{"version":"2","vendors":["ABC.com","XYZ.com"]}
{"version":"1","vendors":["ABC.com"]}
{"version":"2","vendors":["ABC.com","XYZ.com"]}
{"version":"1","vendors":["ABC.com"]}
{"version":"1","vendors":["ABC.com"]}
{"version":"2","vendors":["ABC.com","XYZ.com"]}
{"version":"2","vendors":["ABC.com","XYZ.com"]}