k8s部署rabbitmq集群（使用statefulset部署）

本例使用Storage Class(存储类)来作为集群数据的持久化后端，使用nfs作为后端存储。

1.创建namespace；

# ns.yaml
---
apiVersion: v1
kind: Namespace
metadata:
  name: rabbit-mq

2.创建Service Account为Pod中的程序提供访问Kubernetes API的身份标识，并创建相关的role和rolebinding对Kubernetes的endpoint资源做授权；

# rbac.yaml
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: rabbitmq
  namespace: rabbit-mq
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: endpoint-reader
  namespace: rabbit-mq
rules:
- apiGroups: [""]
  resources: ["endpoints"]
  verbs: ["get"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: endpoint-reader
  namespace: rabbit-mq
subjects:
- kind: ServiceAccount
  name: rabbitmq
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: endpoint-reader

3.创建service访问rabbitmq集群;

# service.yaml
---
kind: Service
apiVersion: v1
metadata:
  name: rabbitmq-headless
  namespace: rabbit-mq
spec:
  clusterIP: None
  publishNotReadyAddresses: true
  ports:
   - name: amqp
     port: 5672
   - name: http
     port: 15672
  selector:
    app: rabbitmq
---
kind: Service
apiVersion: v1
metadata:
  namespace: rabbit-mq
  name: rabbitmq-service
spec:
  ports:
  - name: http
    protocol: TCP
    port: 15672
    nodePort: 30002  #管理web界面
  - name: amqp
    protocol: TCP
    port: 5672
    targetPort: 5672
    nodePort: 30001
  selector:
    app: rabbitmq
  type: NodePort

4.创建secret对象，用来存储rabbitmq的用户名、密码及erlang.cookie；

用户名：admin、密码：123456

# devsecret.yaml
---
apiVersion: v1
kind: Secret
metadata:
  name: devsecret
  namespace: rabbit-mq
type: Opaque
data:
  rabbitDefaulUser: "YWRtaW4="  
  rabbitDefaultPass: "MTIzNDU2"
  erlang.cookie: "di9zV0N6NHVLRVRVdm5lUnlKVm44N0pnMTVzaTJlR2FXZzU0WXZlZmhyaz0="

5.创建Configmap调用rabbitmq配置文件；

# config.yaml
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: rabbitmq-config
  namespace: rabbit-mq
data:
  enabled_plugins: |
      [rabbitmq_management,rabbitmq_peer_discovery_k8s].
  rabbitmq.conf: |
      cluster_formation.peer_discovery_backend  = rabbit_peer_discovery_k8s
      cluster_formation.k8s.host = kubernetes.default.svc.cluster.local
      cluster_formation.k8s.address_type = hostname
      cluster_formation.k8s.hostname_suffix = .rabbitmq-headless.rabbit-mq.svc.cluster.local
      cluster_formation.node_cleanup.interval = 30
      cluster_formation.node_cleanup.only_log_warning = true
      cluster_partition_handling = autoheal
      queue_master_locator=min-masters
      cluster_formation.randomized_startup_delay_range.max = 2
      vm_memory_high_watermark.absolute = 1GB
      disk_free_limit.absolute = 2GB
      loopback_users.guest = false

6.创建statefulset类型构建rabbitmq集群；

存储类名称需要改为自己创建的（storageClassName: "managed-nfs-storage"）

# statefulset.yaml
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: rabbitmq
  namespace: rabbit-mq
spec:
  serviceName: rabbitmq-headless
  selector:
    matchLabels:
      app: rabbitmq  
  replicas: 3  #副本数3
  template:
    metadata:
      labels:
        app: rabbitmq
      annotations:
        scheduler.alpha.kubernetes.io/affinity: >
            {
              "podAntiAffinity": {
                "requiredDuringSchedulingIgnoredDuringExecution": [{
                  "labelSelector": {
                    "matchExpressions": [{
                      "key": "app",
                      "operator": "In",
                      "values": ["rabbitmq"]
                    }]
                  },
                  "topologyKey": "kubernetes.io/hostname"
                }]
              }
            }
    spec:
      serviceAccountName: rabbitmq
      terminationGracePeriodSeconds: 10
      containers:
      - name: rabbitmq
        image: rabbitmq:3.7-rc-management
        resources:
          limits:
            cpu: 2
            memory: 2Gi
          requests:
            cpu: 1
            memory: 1Gi
        volumeMounts:
          - name: config-volume
            mountPath: /etc/rabbitmq
          - name: rabbitmq-data
            mountPath: /var/lib/rabbitmq/mnesia
        ports:
          - name: http
            protocol: TCP
            containerPort: 15672
          - name: amqp
            protocol: TCP
            containerPort: 5672
        livenessProbe:
          exec:
            command: ["rabbitmq-diagnostics", "status"]
          initialDelaySeconds: 60
          periodSeconds: 60
          timeoutSeconds: 5
        readinessProbe:
          exec:
            command: ["rabbitmq-diagnostics", "status"]
          initialDelaySeconds: 20
          periodSeconds: 60
          timeoutSeconds: 5
        imagePullPolicy: Always
        env:
        - name: RABBITMQ_DEFAULT_USER
          valueFrom:
            secretKeyRef:
              key: rabbitDefaulUser
              name: devsecret                
        - name: RABBITMQ_DEFAULT_PASS
          valueFrom:
            secretKeyRef:
              key: rabbitDefaultPass
              name: devsecret
        - name: RABBITMQ_ERLANG_COOKIE
          valueFrom:
            secretKeyRef:
              name: devsecret
              key: erlang.cookie
        - name: HOSTNAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: MY_POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        - name: RABBITMQ_USE_LONGNAME
          value: "true"
        - name: K8S_SERVICE_NAME
          value: "rabbitmq-headless"
        - name: RABBITMQ_NODENAME
          value: rabbit@$(HOSTNAME).$(K8S_SERVICE_NAME).$(MY_POD_NAMESPACE).svc.cluster.local
        - name: K8S_HOSTNAME_SUFFIX
          value: .$(K8S_SERVICE_NAME).$(MY_POD_NAMESPACE).svc.cluster.local
      volumes:
      - name: config-volume
        configMap:
          name: rabbitmq-config
          items:
          - key: rabbitmq.conf
            path: rabbitmq.conf
          - key: enabled_plugins
            path: enabled_plugins
  volumeClaimTemplates:
  - metadata:
      name: rabbitmq-data
    spec:
      storageClassName: "managed-nfs-storage"
      accessModes: [ "ReadWriteMany" ]
      resources:
        requests:
          storage: 2Gi

7.稍微等待一段时间，节点会自动加入集群，不用手动加入。

(1)进入某个节点查看集群状态，如进入rabbitmq-0节点：

kubectl exec -it rabbitmq-0 -n rabbit-mq /bin/bash

(2)进入RabbitMQ Management管理界面查看集群状态：

http://nodeip:30002