k8s service type_RabbitMQ(k8s)-随手笔记

介绍

基于Erlang开发消息队列(AMQP)

在分布式系统中, 有一些功能我们希望能够提高系统稳定性, 比如说支付、订单功能, 服务后移, 长时间操作的功能, 同步数据

我们通过监听数据变化实现功能联动

特点

分布式

各节点互相冗余元数据(erlang.cookie , 队列、交换机、绑定元数据、vhost元数据)实现分布式集群(类似于session), 通过转发达到集群任意节点进入都是集群主节点

可以任意增加节点实现集群水平扩容,但是如果集群是多主节点是不能任意删除节点, 因为实际数据还是存在于各个主节点中, 但是我们可以使用镜像策略来解决这个问题

# 注意使用正则表达式匹配vhost目录
$ rabbitmqctl set_policy ha-all "^ha." '{"ha-mode":"all"}'

可持久化

// 投递消息时设置持久化
channel.queueDeclare("test", true, false, false, null)
// 投递模式设置为2
channel.basicPublish(x, x, MessageProperties.PERSISTENT_TEXT_PLAIN,x)

原理

发送消息: Client -> Exchange -Binding-> Queue -> Server

接收消息: Server -> Channe(线程池) -> Client

Binding可用模式为:

1、Direct, 完全匹配绑定的Queue的名称
2、fanout, 所有Queue
3、topic, 适用于通配符匹配的Queue

Message由Header(属性集合)和Body组成

可以通过Vhost去实现多租户(权限)

客户端可以通过设置AutoACK=false去手动应答以实现更加稳定的数据处理请求, 例如推送到死信队列

对比

相比于Kafka, RabbitMQ更加稳定, 适用于复杂业务场景而非数据传递

相比于ActiveMQ,RabbitMQ更高性能更丰富, 更适用于云原生

相比于RocketMQ, 更加国际化, 更加方便

在K8S中搭建RabbitMQ

参考 https://github.com/rabbitmq/rabbitmq-peer-discovery-k8s

核心基于Kubernetes Discovery 插件(rabbitmq_peer_discovery_k8s)

在使用 StatefulSet 时使用 主机 访问即可

Namespace

apiVersion: v1
kind: Namespace
metadata:
  name: middleware
  labels:
    name: middleware

ConfigMap

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: rabbitmq-config
  namespace: middleware
data:
  enabled_plugins: |
    [rabbitmq_management,rabbitmq_peer_discovery_k8s,rabbitmq_shovel,rabbitmq_shovel_management,accept,prometheus,prometheus_httpd,prometheus_rabbitmq_exporter,prometheus_process_collector].
  rabbitmq.conf: |
    cluster_formation.peer_discovery_backend  = rabbit_peer_discovery_k8s
    cluster_formation.k8s.host = kubernetes.default.svc.cluster.local
    cluster_formation.k8s.address_type = hostname
    cluster_formation.node_cleanup.interval = 10
    cluster_formation.node_cleanup.only_log_warning = true
    cluster_partition_handling = autoheal
    queue_master_locator=min-masters
    loopback_users.guest = false
    cluster_formation.randomized_startup_delay_range.min = 0
    cluster_formation.randomized_startup_delay_range.max = 2
    cluster_formation.k8s.service_name = rabbitmq-headless
    cluster_formation.k8s.hostname_suffix = .rabbitmq-headless.rabbitmq.svc.cluster.local
    vm_memory_high_watermark.absolute = 3.2GB
    disk_free_limit.absolute = 10GB
    default_user = test
    default_pass = test

配置参数讲解:

| 配置 | 含义 | | :----------------------------------------------------------- | :----------------------------------------------------------: | | cluster_formation.peer_discovery_backend = rabbit_peer_discovery_k8s | 指定集群节点发现机制 | | cluster_formation.k8s.host = kubernetes.default.svc.cluster.local | Kubernetes API 主机名 | | cluster_formation.k8s.address_type = hostname | 访问其他节点时使用的方式 | | cluster_formation.node_cleanup.interval = 10 | 每隔多久清理上次获取的节点主机并获取新的节点主机 | | cluster_formation.node_cleanup.only_log_warning = true | 记录日志 | | cluster_partition_handling = autoheal | 网络异常时, 使用网络分区将异常的节点排离在分区之外, 这里设置为自动决定一个获胜的（winning）分区，然后重启不在这个分区中的节点以恢复网络分区。一个获胜的分区是指客户端连接最多的一个分区。如果产生一个平局，既有两个或者多个分区的客户端连接数一样多，那么节点数最多的一个分区就是获胜的分区。如果此时节点数也一样多，将会以一种特殊的方式来挑选获胜分区 | | queue_master_locator=min-masters | 队列主节点的策略为选择托管最少数量的绑定主节点的节点 | | cluster_formation.randomized_startup_delay_range.min = 0 cluster_formation.randomized_startup_delay_range.max = 2 | 随机延迟初始化时间, 单位为秒 | | cluster_formation.k8s.service_name = rabbitmq-headless | 对应部署对象的K8S_SERVICE_NAME变量 | | cluster_formation.k8s.hostname_suffix = .rabbitmq-headless.rabbitmq.svc.cluster.local | 集群访问的后缀 | | vm_memory_high_watermark.absolute = 3.2GB | 内存上限 | | disk_free_limit.absolute = 10GB | 磁盘上限 | | default_user = test default_pass = test | 默认初始的账号及其密码 | | | |

Service

---
kind: Service
apiVersion: v1
metadata:
  name: rabbitmq-headless
  namespace: middleware
spec:
  clusterIP: None
  publishNotReadyAddresses: true
  ports:
    - name: amqp
      port: 5672
    - name: http
      port: 15672
  selector:
    app: rabbitmq
---
kind: Service
apiVersion: v1
metadata:
  namespace: middleware
  name: rabbitmq-service
spec:
  type: NodePort
  ports:
    - name: http
      protocol: TCP
      port: 15672
      targetPort: 15672
    - name: amqp
      protocol: TCP
      port: 5672
  selector:
    app: rabbitmq

RBAC

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: rabbitmq
  namespace: middleware
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: endpoint-reader
  namespace: middleware
rules:
  - apiGroups: [""]
    resources: ["endpoints"]
    verbs: ["get"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: endpoint-reader
  namespace: middleware
subjects:
  - kind: ServiceAccount
    name: rabbitmq
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: endpoint-reader

StatefulSet

---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: rabbitmq
  namespace: middleware
spec:
  serviceName: rabbitmq-headless
  selector:
    matchLabels:
      app: rabbitmq
  replicas: 1
  template:
    metadata:
      labels:
        app: rabbitmq
      annotations:
        scheduler.alpha.kubernetes.io/affinity: >
          {
            "podAntiAffinity": {
              "requiredDuringSchedulingIgnoredDuringExecution": [{
                "labelSelector": {
                  "matchExpressions": [{
                    "key": "app",
                    "operator": "In",
                    "values": ["rabbitmq"]
                  }]
                },
                "topologyKey": "kubernetes.io/hostname"
              }]
            }
          }
    spec:
      nodeSelector:
        middleware: "middleware"
      serviceAccountName: rabbitmq
      terminationGracePeriodSeconds: 10
      containers:
        - name: rabbitmq
          image: registry-vpc.cn-shenzhen.aliyuncs.com/heygears/rabbitmq:3.7
          resources:
            limits:
              cpu: 0.5
              memory: 2Gi
            requests:
              cpu: 0.3
              memory: 2Gi
          volumeMounts:
            - name: config-volume
              mountPath: /etc/rabbitmq
              readOnly: true
            - name: rabbitmq-data
              mountPath: /var/lib/rabbitmq/mnesia
              subPath: rabbitmq-ha
          ports:
            - name: http
              protocol: TCP
              containerPort: 15672
            - name: amqp
              protocol: TCP
              containerPort: 5672
          livenessProbe:
            exec:
              command: ["rabbitmqctl", "status"]
            initialDelaySeconds: 60
            periodSeconds: 60
            timeoutSeconds: 5
          readinessProbe:
            exec:
              command: ["rabbitmqctl", "status"]
            initialDelaySeconds: 20
            periodSeconds: 60
            timeoutSeconds: 5
          imagePullPolicy: Always
          env:
            - name: HOSTNAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: RABBITMQ_USE_LONGNAME
              value: "true"
            - name: RABBITMQ_NODENAME
              value: "rabbit@$(HOSTNAME).rabbitmq-headless.middleware.svc.cluster.local"
            - name: RABBITMQ_ERLANG_COOKIE
              value: "mycookie"
      volumes:
        - name: config-volume
          configMap:
            name: rabbitmq-config
            items:
              - key: rabbitmq.conf
                path: rabbitmq.conf
              - key: enabled_plugins
                path: enabled_plugins
        - name: rabbitmq-data
          persistentVolumeClaim:
            claimName: nfs-pvc-middleware

PV/PVC

---
apiVersion: v1
kind: PersistentVolume
metadata:
  namespace: middleware
  name: nfs-pv-middleware
  labels:
    app: nfs
spec:
  capacity:
    storage: 500Gi
  accessModes:
    - ReadWriteMany
  mountOptions:
    - vers=3
    - nolock,tcp,noresvport
  nfs:
    path: /
    server: xxx.nas.aliyuncs.com

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  namespace: middleware
  name: nfs-pvc-middleware
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 100Gi
  selector:
    matchLabels:
      app: nfs

IngressRoute

---
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: rabbitmq-ingress-route
  namespace: middleware
spec:
  entryPoints:
    - web
  routes:
    - match: Host(`xxx.com`) && PathPrefix(`/`)
      kind: Rule
      services:
        - name: rabbitmq-service
          port: 15672