k8s1.20版本部署RabbitMQ集群(持久化)——2023.05

一、集群概况

1. 主机规划

节点	IP
k8s-master1	192.168.2.245
k8s-master2	192.168.2.246
k8s-master3	192.168.2.247
k8s-node1	192.168.2.248
NFS、Rancher	192.168.2.251

2. 版本介绍

服务	版本
centos	7.9
Rancher（单节点）	2.5.12
kubernetes	1.20.15
RabbitMQ	3.7-management

二、RabbitMQ集群部署

2.1 安装NFS

NFS Server IP（服务端）：192.168.2.251
NFS Client IP（客户端）：192.168.2.245

1. NFS Server端安装NFS
操作主机：NFS、Rancher|192.168.2.251

# 1.安装nfs与rpc
yum install -y nfs-utils rpcbind
# 查看是否安装成功
rpm -qa | grep nfs
rpm -qa | grep rpcbind

# 2.创建共享存储文件夹
mkdir -p /k8s/rmq-cluster 

# 3.配置nfs
vim /etc/exports
/k8s/rmq-cluster 192.168.2.0/24(rw,no_root_squash,no_all_squash,sync)

# 4.启动服务
systemctl start nfs
systemctl start rpcbind
#添加开机自启
systemctl enable nfs
systemctl enable rpcbind

# 5.配置生效
exportfs -r

# 6.查看挂载情况
showmount -e localhost
#输出下面信息表示正常
Export list for localhost:
/nfs/k8s_data 192.168.2.0/24

2. NFS Client安装NFS
   操作主机：除了NFS server，其他所有主机

yum -y install nfs-utils

2.2 创建storageclass存储类

1. 创建ServiceAccount账号

vim rabbitmq-rbac.yaml
#复制以下内容：

apiVersion: v1
kind: ServiceAccount
metadata:
  name: rmq-cluster
  namespace: public-service
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: rmq-cluster
  namespace: public-service
rules:
  - apiGroups: [""]
    resources: ["endpoints"]
    verbs: ["get"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: rmq-cluster
  namespace: public-service
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: rmq-cluster
subjects:
- kind: ServiceAccount
  name: rmq-cluster
  namespace: public-service

# 创建资源
kubectl create -f rabbitmq-rbac.yaml

2. 创建provisioner
（也可称为供应者、置备程序、存储分配器）

vim nfs-client-provisioner.yaml
# 复制以下内容：

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nfs-client-provisioner
  labels:
    app: nfs-client-provisioner
  namespace: public-service
spec:
  replicas: 1
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app: nfs-client-provisioner
  template:
    metadata:
      labels:
        app: nfs-client-provisioner
    spec:
      serviceAccountName: rmq-cluster                                #这个serviceAccountName就是上面创建ServiceAccount账号
      containers:
        - name: nfs-client-provisioner
          image: quay.io/external_storage/nfs-client-provisioner:latest
          volumeMounts:
            - name: nfs-client-root
              mountPath: /persistentvolumes
          env:
            - name: PROVISIONER_NAME                    #PROVISIONER_NAME的值就是本清单的顶部定义的name
              value: nfs-client-provisioner
            - name: NFS_SERVER                                  #这个NFS_SERVER参数的值就是nfs服务器的IP地址
              value: 192.168.2.251
            - name: NFS_PATH                                    #这个NFS_PATH参数的值就是nfs服务器的共享目录
              value: /k8s/rmq-cluster
      volumes:
        - name: nfs-client-root
          nfs:                                                                  #这里就是配置nfs服务器的ip地址和共享目录
            server: 192.168.2.251
            path: /k8s/rmq-cluster

# 创建资源
kubectl create -f nfs-client-provisioner.yaml

3. 创建StorageClass

vim nfs-storageclass.yaml
# 复制以下内容：

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: rabbitmq-storageclass
parameters:
  archiveOnDelete: 'false'
provisioner: nfs-client-provisioner     # 与provisioner的name一致
reclaimPolicy: Delete
volumeBindingMode: Immediate

# 创建资源：
kubectl create -f nfs-storageclass.yaml	

2.3 部署RabbitMQ集群

1. 创建命名空间

kubectl create namespace public-service

2. 创建访问rabbitmq集群的service和屋头service

vim rabbitmq-service-lb.yaml
复制以下内容：

kind: Service
apiVersion: v1
metadata:
  labels:
    app: rmq-cluster
    type: LoadBalancer
  name: rmq-cluster-balancer
  namespace: public-service
spec:
  ports:
  - name: http
    port: 15672
    protocol: TCP
    targetPort: 15672
  - name: amqp
    port: 5672
    protocol: TCP
    targetPort: 5672
  selector:
    app: rmq-cluster
  type: NodePort

vim rabbitmq-service-cluster.yaml
复制以下内容：

apiVersion: v1
metadata:
  labels:
    app: rmq-cluster
  name: rmq-cluster
  namespace: public-service
spec:
  clusterIP: None
  ports:
  - name: amqp
    port: 5672
    targetPort: 5672
  selector:
    app: rmq-cluster

# 创建资源
kubectl create -f  rabbitmq-service-lb.yaml rabbitmq-service-cluster.yaml 

3. 创建一个secret对象，用来存储rabbitmq的用户名、密码

vim rabbitmq-secret.yaml
复制以下内容：

kind: Secret
apiVersion: v1
metadata:
  name: rmq-cluster-secret
  namespace: public-service
stringData:
  cookie: ERLANG_COOKIE
  password: RABBITMQ_PASS
  url: amqp://RABBITMQ_USER:RABBITMQ_PASS@rmq-cluster-balancer
  username: RABBITMQ_USER
type: Opaque

# 创建资源：
kubectl apply -f rabbitmq-secret.yaml

4. Configmap创建 rabbitmq配置文件

vim rabbitmq-configmap.yaml
复制以下内容：

kind: ConfigMap
apiVersion: v1
metadata:
  name: rmq-cluster-config
  namespace: public-service
  labels:
    addonmanager.kubernetes.io/mode: Reconcile
data:
    enabled_plugins: |
      [rabbitmq_management,rabbitmq_peer_discovery_k8s].
    rabbitmq.conf: |
      loopback_users.guest = false

      default_user = RABBITMQ_USER
      default_pass = RABBITMQ_PASS
      ## Clustering
      cluster_formation.peer_discovery_backend = rabbit_peer_discovery_k8s
      cluster_formation.k8s.host = kubernetes.default.svc.cluster.local
      cluster_formation.k8s.address_type = hostname
      #################################################
      # public-service is rabbitmq-cluster's namespace#
      #################################################
      cluster_formation.k8s.hostname_suffix = .rmq-cluster.public-service.svc.cluster.local
      cluster_formation.node_cleanup.interval = 10
      cluster_formation.node_cleanup.only_log_warning = true
      cluster_partition_handling = autoheal
      ## queue master locator
      queue_master_locator=min-masters

# 创建资源
kubectl apply -f rabbitmq-configmap.yaml

5. 通过statefulset类型创建rabbitmq集群

vim rabbitmq-cluster-sts.yaml
复制以下内容：

kind: StatefulSet
apiVersion: apps/v1
metadata:
  labels:
    app: rmq-cluster
  name: rmq-cluster
  namespace: public-service
spec:
  replicas: 3
  selector:
    matchLabels:
      app: rmq-cluster
  serviceName: rmq-cluster
  template:
    metadata:
      labels:
        app: rmq-cluster
    spec:
      containers:
      - args:
        - -c
        - cp -v /etc/rabbitmq/rabbitmq.conf ${RABBITMQ_CONFIG_FILE}; exec docker-entrypoint.sh
          rabbitmq-server
        command:
        - sh
        env:      
        - name: RABBITMQ_DEFAULT_USER     #登陆用户名和密码都存储在一个secret对象中
          valueFrom:
            secretKeyRef:
              key: username
              name: rmq-cluster-secret
        - name: RABBITMQ_DEFAULT_PASS
          valueFrom:
            secretKeyRef:
              key: password
              name: rmq-cluster-secret
        - name: RABBITMQ_ERLANG_COOKIE
          valueFrom:
            secretKeyRef:
              key: cookie
              name: rmq-cluster-secret
        - name: K8S_SERVICE_NAME
          value: rmq-cluster
        - name: POD_IP
          valueFrom:
            fieldRef:
              fieldPath: status.podIP
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        - name: RABBITMQ_USE_LONGNAME
          value: "true"
        - name: RABBITMQ_NODENAME
          value: rabbit@$(POD_NAME).rmq-cluster.$(POD_NAMESPACE).svc.cluster.local
        - name: RABBITMQ_CONFIG_FILE
          value: /var/lib/rabbitmq/rabbitmq.conf
        image: registry.cn-hongkong.aliyuncs.com/susie/rabbitmq:3.7-management
        imagePullPolicy: IfNotPresent
        livenessProbe:
          exec:
            command:
            - rabbitmqctl
            - status
          initialDelaySeconds: 30
          timeoutSeconds: 10
        name: rabbitmq
        ports:
        - containerPort: 15672
          name: http
          protocol: TCP
        - containerPort: 5672
          name: amqp
          protocol: TCP
        readinessProbe:
          exec:
            command:
            - rabbitmqctl
            - status
          initialDelaySeconds: 10
          timeoutSeconds: 10
        volumeMounts:
        - mountPath: /etc/rabbitmq
          name: config-volume
          readOnly: false
#        - mountPath: /var/lib/rabbitmq
#          name: rabbitmq-storage
#          readOnly: false
      serviceAccountName: rmq-cluster
      terminationGracePeriodSeconds: 30
      volumes:
      - configMap:
          items:
          - key: rabbitmq.conf
            path: rabbitmq.conf
          - key: enabled_plugins
            path: enabled_plugins
          name: rmq-cluster-config
        name: config-volume
  volumeClaimTemplates:
  - metadata:
      name: rabbitmq-storage
    spec:
      accessModes:
      - ReadWriteMany
      storageClassName: "rabbitmq-storage-class"    # 配置前面创建好的storageclass名称
      resources:
        requests:
          storage: 1Gi    #设置大小

# 创建资源：
kubectl apply -f rabbitmq-cluster-sts.yaml

查看资源创建情况：
kubectl get pod,svc,pv,pvc -n public-service

可以看到3个pod 已经running，并且pvc 均已自动绑定。

2.4 测试

1. 进入一个pod，查看rabbitmq集群状态：

[root@k8s-master1 rabbitmq]# kubectl exec -it rmq-cluster-0 -n public-service bash
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.

root@rmq-cluster-0:/# rabbitmqctl cluster_status
Cluster status of node rabbit@rmq-cluster-0.rmq-cluster.public-service.svc.cluster.local ...
[{nodes,
     [{disc,
          ['rabbit@rmq-cluster-0.rmq-cluster.public-service.svc.cluster.local',
           'rabbit@rmq-cluster-1.rmq-cluster.public-service.svc.cluster.local',
           'rabbit@rmq-cluster-2.rmq-cluster.public-service.svc.cluster.local']}]},
 {running_nodes,
     ['rabbit@rmq-cluster-2.rmq-cluster.public-service.svc.cluster.local',
      'rabbit@rmq-cluster-1.rmq-cluster.public-service.svc.cluster.local',
      'rabbit@rmq-cluster-0.rmq-cluster.public-service.svc.cluster.local']},
 {cluster_name,
     <<"rabbit@rmq-cluster-0.rmq-cluster.public-service.svc.cluster.local">>},
 {partitions,[]},
 {alarms,
     [{'rabbit@rmq-cluster-2.rmq-cluster.public-service.svc.cluster.local',[]},
      {'rabbit@rmq-cluster-1.rmq-cluster.public-service.svc.cluster.local',[]},
      {'rabbit@rmq-cluster-0.rmq-cluster.public-service.svc.cluster.local',
          []}]}]

2. RabbitMQ Management 管理界面查看集群状态
http://nodeip:service暴露的IP登录管理界面，这里是http://192.168.2.248:31149/

部署完成。

参考文章：
k8s部署rabbitMQ集群