基于k8s部署rocketmq两主两从集群

本次部署为基于k8s部署rocketmq两主两从集群,本次服务器为:

1.设置nfs存储

(1) yum下载nfs,rpcbind,四台机器上都执行

yum install -y nfs-utils rpcbind

(2) 在176上执行,设置/data为nfs共享目录，在exports文件中新增下列内容

vi /etc/exports

/data/ *(rw,sync,no_root_squash)
(3) 176节点上执行,创建数据存储路径

mkdir -p /data/rocketmq

(4) 176节点上执行,设置数据存储路径文件权限

chmod -R 755 /data/rocketmq

(5) 176节点上执行,使新增的nfs共享目录生效

exportfs -arv

(6) 四台机器上都执行,启动rpcbind并设置开机自启

systemctl enable rpcbind && systemctl start rpcbind

(7) 四台机器上都执行,启动nfs并设置开机自启

systemctl enable nfs && systemctl start nfs

2.创建nfs相关pod.

176上以root用户操作,当前操作路径为/root/rocketmq,之后操作都在次路径下.

(1) 创建rabc

vim nfs-raabc.yaml

apiVersion: v1

kind: ServiceAccount

metadata:

  name: nfs-provisioner

  namespace: wiseco

---

kind: ClusterRole

apiVersion: rbac.authorization.k8s.io/v1

metadata:

   name: nfs-provisioner-runner

   namespace: wiseco

rules:

   -  apiGroups: [""]

      resources: ["persistentvolumes"]

      verbs: ["get", "list", "watch", "create", "delete"]

   -  apiGroups: [""]

      resources: ["persistentvolumeclaims"]

      verbs: ["get", "list", "watch", "update"]

   -  apiGroups: ["storage.k8s.io"]

      resources: ["storageclasses"]

      verbs: ["get", "list", "watch"]

   -  apiGroups: [""]

      resources: ["events"]

      verbs: ["watch", "create", "update", "patch"]

   -  apiGroups: [""]

      resources: ["services", "endpoints"]

      verbs: ["get","create","list", "watch","update"]

   -  apiGroups: ["extensions"]

      resources: ["podsecuritypolicies"]

      resourceNames: ["nfs-provisioner"]

      verbs: ["use"]

---

kind: ClusterRoleBinding

apiVersion: rbac.authorization.k8s.io/v1

metadata:

  name: run-nfs-provisioner

subjects:

  - kind: ServiceAccount

    name: nfs-provisioner

    namespace: wiseco

roleRef:

  kind: ClusterRole

  name: nfs-provisioner-runner

  apiGroup: rbac.authorization.k8s.io

(2) 创建storageclass

vim rocketmq-nfs-class.yaml

kind: StorageClass

metadata:

  name: rocketmq-nfs-storage

  namespace: wiseco

provisioner: rocketmq/nfs

reclaimPolicy: Retain

(3) 创建rocketmq-nfs-class.yaml

vim rocketmq-nfs.yml

kind: Deployment

metadata:

  name: rocketmq-nfs-client-provisioner

  namespace: wiseco

spec:

  replicas: 1

  selector:

    matchLabels:

      app: rocketmq-nfs-client-provisioner

  strategy:

    type: Recreate

  template:

    metadata:

      labels:

        app: rocketmq-nfs-client-provisioner

    spec:

      serviceAccount: nfs-provisioner

      containers:

        - name: rocketmq-nfs-client-provisioner

          image: registry.cn-hangzhou.aliyuncs.com/open-ali/nfs-client-provisioner

          imagePullPolicy: IfNotPresent

          volumeMounts:

            - name: nfs-client-root

              mountPath:  /persistentvolumes

          env:

            - name: PROVISIONER_NAME

              value: rocketmq/nfs

            - name: NFS_SERVER

              value: 192.168.212.176

            - name: NFS_PATH

              value: /data/rocketmq

      volumes:

        - name: nfs-client-root

          nfs:

            server: 192.168.212.176

            path: /data/rocketmq

注:192.168.212.176更改为自己的ip

(4) 使上面三个配置文件生效

① kubectl apply -f nfs-raabc.yaml (kubectl apply -f 可以理解为使配置文件生效,这里就是使nfs-raabc.yaml文件生效)

② kubectl apply -f rocketmq-nfs-class.yaml

③ kubectl apply -f rocketmq-nfs.yml

(5) 查看rocketmq的nfs是否创建成功

kubectl get pods -n wiseco|grep nfs

3.制作镜像

镜像已制作好,把这三个镜像包包分别上传到176,177,178,179四台机器上并使用docker load -i 命令转为镜像即可.镜像制作看查看参考文档.

4.准备配置文件,创建configmap

进入config目录,需要自己创建并进入,编辑四个broker配置文件
(1) vim broker-a.properties

brokerClusterName=rocketmq-cluster

brokerName=broker-a

brokerIP1=192.168.212.178

brokerId=0

namesrvAddr=192.168.212.179:30915

defaultTopicQueueNums=4

autoCreateTopicEnable=true

autoCreateSubscriptionGroup=true

listenPort=30911

deleteWhen=04

fileReservedTime=120

mapedFileSizeCommitLog=1073741824

mapedFileSizeConsumeQueue=300000

diskMaxUsedSpaceRatio=88

storePathRootDir=/data/rocketmq/store

maxMessageSize=65536

brokerRole=MASTER

(2) vim broker-a-s.properties

brokerClusterName=rocketmq-cluster

brokerName=broker-a

brokerIP1=192.168.212.179

brokerId=1

namesrvAddr=192.168.212.179:30915

defaultTopicQueueNums=4

autoCreateTopicEnable=true

autoCreateSubscriptionGroup=true

listenPort=30913

deleteWhen=04

fileReservedTime=120

mapedFileSizeCommitLog=1073741824

mapedFileSizeConsumeQueue=300000

diskMaxUsedSpaceRatio=88

storePathRootDir=/data/rocketmq/store

maxMessageSize=65536

brokerRole=SLAVE

flushDiskType=SYNC_FLUSH

(3) vim broker-b.properties

brokerClusterName=rocketmq-cluster

brokerName=broker-b

brokerIP1=192.168.212.179

brokerId=0

namesrvAddr=192.168.212.179:30915

defaultTopicQueueNums=4

autoCreateTopicEnable=true

autoCreateSubscriptionGroup=true

listenPort=30912

deleteWhen=04

fileReservedTime=120

mapedFileSizeCommitLog=1073741824

mapedFileSizeConsumeQueue=300000

diskMaxUsedSpaceRatio=88

storePathRootDir=/data/rocketmq/store

maxMessageSize=65536

brokerRole=MASTER

flushDiskType=SYNC_FLUSH

(4) vim broker-b-s.properties

brokerClusterName=rocketmq-cluster

brokerName=broker-b

brokerIP1=192.168.212.178

brokerId=1

namesrvAddr=192.168.212.179:30915

defaultTopicQueueNums=4

autoCreateTopicEnable=true

autoCreateSubscriptionGroup=true

listenPort=30914

deleteWhen=04

fileReservedTime=120

mapedFileSizeCommitLog=1073741824

mapedFileSizeConsumeQueue=300000

diskMaxUsedSpaceRatio=88

storePathRootDir=/data/rocketmq/store

maxMessageSize=65536

brokerRole=SLAVE

flushDiskType=SYNC_FLUSH

(5) 依据上面rocketmq的4个配置文件创建configmap存储卷

kubectl create configmap rocketmq-config --from-file=broker-a.properties --from-file=broker-b.properties --from-file=broker-a-s.properties --from-file=broker-b-s.properties -n wiseco (根据四个broker创建configmap)

(6) 查询配置文件是否生成

kubectl get cm -n wiseco|grep rocketmq

5.部署rocketmq容器实例

在rocketmq目录下新建deploy目录并进入

(1) vim broker-a-deployment.yaml

apiVersion: v1

kind: Service

metadata:

  labels:

    app: broker-a

  name: broker-a

  namespace: wiseco

spec:

  type: NodePort

  ports:

  - port: 30911

    targetPort: 30911

    name: broker-port

    nodePort: 30911

  selector:

    app: broker-a

---

apiVersion: apps/v1

kind: StatefulSet

metadata:

  name: broker-a

  namespace: wiseco

spec:

  serviceName: broker-a

  replicas: 1

  selector:

    matchLabels:

      app: broker-a

  template:

    metadata:

     labels:

       app: broker-a

    spec:

      affinity:

        podAntiAffinity:

          requiredDuringSchedulingIgnoredDuringExecution:

            - labelSelector:

                matchExpressions:

                  - key: "app"

                    operator: In

                    values:

                      - broker-a

              topologyKey: "kubernetes.io/hostname"

      containers:

      - name: broker-a

        image: wiseco/rocketmq:v1

        imagePullPolicy: Never

        command: ["sh","-c","/usr/local/rocketmq-4.9.1/bin/mqbroker  -c /usr/local/rocketmq-4.9.1/conf/broker-a.properties"]

        volumeMounts:

          - mountPath: /root/logs

            name: rocketmq-data

            subPath: mq-brokeroptlogs

          - mountPath: /data/rocketmq

            name: rocketmq-data

            subPath: mq-brokeroptstore

          - name: broker-config

            mountPath: /usr/local/rocketmq-4.9.1/conf/broker-a.properties

            subPath: broker-a.properties

        lifecycle:

          postStart:

            exec:

              command: ["/bin/sh","-c","touch /tmp/health"]

        livenessProbe:

          exec:

            command: ["test","-e","/tmp/health"]

          initialDelaySeconds: 5

          timeoutSeconds: 5

          periodSeconds: 10

        readinessProbe:

          tcpSocket:

            port: 30911

          initialDelaySeconds: 15

          timeoutSeconds: 5

          periodSeconds: 20

      volumes:

      - name: broker-config

        configMap:

          name: rocketmq-config

  volumeClaimTemplates:

  - metadata:

      name: rocketmq-data

      annotations:

        volume.beta.kubernetes.io/storage-class: "rocketmq-nfs-storage"

    spec:

      accessModes:

        - ReadWriteMany

      resources:

        requests:

          storage: 300Mi

(2) vim broker-a-s-deployment.yaml

apiVersion: v1

kind: Service

metadata:

  labels:

    app: broker-a-s

  name: broker-a-s

  namespace: wiseco

spec:

  type: NodePort

  ports:

  - port: 30913

    targetPort: 30913

    name: broker-port

    nodePort: 30913

  selector:

    app: broker-a-s

---

apiVersion: apps/v1

kind: StatefulSet

metadata:

  name: broker-a-s

  namespace: wiseco

spec:

  serviceName: broker-a-s

  replicas: 1

  selector:

    matchLabels:

      app: broker-a-s

  template:

    metadata:

     labels:

       app: broker-a-s

    spec:

      affinity:

        podAntiAffinity:

          requiredDuringSchedulingIgnoredDuringExecution:

            - labelSelector:

                matchExpressions:

                  - key: "app"

                    operator: In

                    values:

                      - broker-a-s

              topologyKey: "kubernetes.io/hostname"

      containers:

      - name: broker-a-s

        image: wiseco/rocketmq:v1

        imagePullPolicy: Never

        command: ["sh","-c","/usr/local/rocketmq-4.9.1/bin/mqbroker  -c /usr/local/rocketmq-4.9.1/conf/broker-a-s.properties"]

        volumeMounts:

          - mountPath: /root/logs

            name: rocketmq-data

            subPath: mq-brokeroptlogs

          - mountPath: /data/rocketmq

            name: rocketmq-data

            subPath: mq-brokeroptstore

          - name: broker-config

            mountPath: /usr/local/rocketmq-4.9.1/conf/broker-a-s.properties

            subPath: broker-a-s.properties

        lifecycle:

          postStart:

            exec:

              command: ["/bin/sh","-c","touch /tmp/health"]

        livenessProbe:

          exec:

            command: ["test","-e","/tmp/health"]

          initialDelaySeconds: 5

          timeoutSeconds: 5

          periodSeconds: 10

        readinessProbe:

          tcpSocket:

            port: 30913

          initialDelaySeconds: 15

          timeoutSeconds: 5

          periodSeconds: 20

      volumes:

      - name: broker-config

        configMap:

          name: rocketmq-config

          items:

          - key: broker-a-s.properties

            path: broker-a-s.properties

  volumeClaimTemplates:

  - metadata:

      name: rocketmq-data

      annotations:

        volume.beta.kubernetes.io/storage-class: "rocketmq-nfs-storage"

    spec:

      accessModes:

        - ReadWriteMany

      resources:

        requests:

          storage: 300Mi

(3) vim broker-b-deployment.yaml

apiVersion: v1

kind: Service

metadata:

  labels:

    app: broker-b

  name: broker-b

  namespace: wiseco

spec:

  type: NodePort

  ports:

  - port: 30912

    targetPort: 30912

    name: broker-port

    nodePort: 30912

  selector:

    app: broker-b

---

apiVersion: apps/v1

kind: StatefulSet

metadata:

  name: broker-b

  namespace: wiseco

spec:

  serviceName: broker-b

  replicas: 1

  selector:

    matchLabels:

      app: broker-b

  template:

    metadata:

     labels:

       app: broker-b

    spec:

      affinity:

        podAntiAffinity:

          requiredDuringSchedulingIgnoredDuringExecution:

            - labelSelector:

                matchExpressions:

                  - key: "app"

                    operator: In

                    values:

                      - broker-b

              topologyKey: "kubernetes.io/hostname"

      containers:

      - name: broker-b

        image: wiseco/rocketmq:v1

        imagePullPolicy: Never

        command: ["sh","-c","/usr/local/rocketmq-4.9.1/bin/mqbroker  -c /usr/local/rocketmq-4.9.1/conf/broker-b.properties"]

        volumeMounts:

          - mountPath: /root/logs

            name: rocketmq-data

            subPath: mq-brokeroptlogs

          - mountPath: /data/rocketmq

            name: rocketmq-data

            subPath: mq-brokeroptstore

          - name: broker-config

            mountPath: /usr/local/rocketmq-4.9.1/conf/broker-b.properties

            subPath: broker-b.properties

        lifecycle:

          postStart:

            exec:

              command: ["/bin/sh","-c","touch /tmp/health"]

        livenessProbe:

          exec:

            command: ["test","-e","/tmp/health"]

          initialDelaySeconds: 5

          timeoutSeconds: 5

          periodSeconds: 10

        readinessProbe:

          tcpSocket:

            port: 30912

          initialDelaySeconds: 15

          timeoutSeconds: 5

          periodSeconds: 20

      volumes:

      - name: broker-config

        configMap:

          name: rocketmq-config

  volumeClaimTemplates:

  - metadata:

      name: rocketmq-data

      annotations:

        volume.beta.kubernetes.io/storage-class: "rocketmq-nfs-storage"

    spec:

      accessModes:

        - ReadWriteMany

      resources:

        requests:

          storage: 300Mi

(4) vim broker-b-s-deployment.yaml

apiVersion: v1

kind: Service

metadata:

  labels:

    app: broker-b-s

  name: broker-b-s

  namespace: wiseco

spec:

  type: NodePort

  ports:

  - port: 30914

    targetPort: 30914

    name: broker-port

    nodePort: 30914

  selector:

    app: broker-b-s

---

apiVersion: apps/v1

kind: StatefulSet

metadata:

  name: broker-b-s

  namespace: wiseco

spec:

  serviceName: broker-b-s

  replicas: 1

  selector:

    matchLabels:

      app: broker-b-s

  template:

    metadata:

     labels:

       app: broker-b-s

    spec:

      affinity:

        podAntiAffinity:

          requiredDuringSchedulingIgnoredDuringExecution:

            - labelSelector:

                matchExpressions:

                  - key: "app"

                    operator: In

                    values:

                      - broker-b-s

              topologyKey: "kubernetes.io/hostname"

      containers:

      - name: broker-b-s

        image: wiseco/rocketmq:v1

        imagePullPolicy: Never

        command: ["sh","-c","/usr/local/rocketmq-4.9.1/bin/mqbroker  -c /usr/local/rocketmq-4.9.1/conf/broker-b-s.properties"]

        volumeMounts:

          - mountPath: /root/logs

            name: rocketmq-data

            subPath: mq-brokeroptlogs

          - mountPath: /data/rocketmq

            name: rocketmq-data

            subPath: mq-brokeroptstore

          - name: broker-config

            mountPath: /usr/local/rocketmq-4.9.1/conf/broker-b-s.properties

            subPath: broker-b-s.properties

        lifecycle:

          postStart:

            exec:

              command: ["/bin/sh","-c","touch /tmp/health"]

        livenessProbe:

          exec:

            command: ["test","-e","/tmp/health"]

          initialDelaySeconds: 5

          timeoutSeconds: 5

          periodSeconds: 10

        readinessProbe:

          tcpSocket:

            port: 30914

          initialDelaySeconds: 15

          timeoutSeconds: 5

          periodSeconds: 20

      volumes:

      - name: broker-config

        configMap:

          name: rocketmq-config

          items:

          - key: broker-b-s.properties

            path: broker-b-s.properties

  volumeClaimTemplates:

  - metadata:

      name: rocketmq-data

      annotations:

        volume.beta.kubernetes.io/storage-class: "rocketmq-nfs-storage"

    spec:

      accessModes:

        - ReadWriteMany

      resources:

        requests:

          storage: 300Mi

(5) vim namesrv-deployment.yaml

apiVersion: v1

kind: Service

metadata:

  labels:

    app: mq-namesrv

  name: mq-namesrv

  namespace: wiseco

spec:

  type: NodePort

  ports:

  - port: 20901

    targetPort: 20901

    name: namesrv-port

    nodePort: 30915

  selector:

    app: mq-namesrv

---

apiVersion: apps/v1

kind: StatefulSet

metadata:

  name: mq-namesrv

  namespace: wiseco

spec:

  serviceName: mq-namesrv

  replicas: 1

  selector:

    matchLabels:

      app: mq-namesrv

  template:

    metadata:

     labels:

       app: mq-namesrv

    spec:

      affinity:

        podAntiAffinity:

          requiredDuringSchedulingIgnoredDuringExecution:

            - labelSelector:

                matchExpressions:

                  - key: "app"

                    operator: In

                    values:

                      - mq-namesrv

              topologyKey: "kubernetes.io/hostname"

      containers:

      - name: mq-namesrv

        image: wiseco/nameserver:v1

        imagePullPolicy: Never

        lifecycle:

          postStart:

            exec:

              command: ["/bin/sh","-c","touch /tmp/health"]

        livenessProbe:

          exec:

            command: ["test","-e","/tmp/health"]

          initialDelaySeconds: 5

          timeoutSeconds: 5

          periodSeconds: 10

        readinessProbe:

          tcpSocket:

            port: 20901

          initialDelaySeconds: 15

          timeoutSeconds: 5

          periodSeconds: 20

(6) vim rocketmq-externals-deployment.yaml

apiVersion: v1

kind: Service

metadata:

  labels:

    app: mq-externals

  name: mq-externals

  namespace: wiseco

spec:

  type: NodePort

  ports:

  - port: 8080

    targetPort: 8080

    name: console-port

    nodePort: 30916

  selector:

    app: mq-externals

---

apiVersion: apps/v1

kind: Deployment

metadata:

  name: mq-externals

  namespace: wiseco

spec:

  replicas: 1

  selector:

    matchLabels:

      app: mq-externals

  template:

    metadata:

     labels:

       app: mq-externals

    spec:

      affinity:

        podAntiAffinity:

          requiredDuringSchedulingIgnoredDuringExecution:

            - labelSelector:

                matchExpressions:

                  - key: "app"

                    operator: In

                    values:

                      - mq-externals

              topologyKey: "kubernetes.io/hostname"

      containers:

      - name: mq-externals

        image: wiseco/mqexternals:v1

        imagePullPolicy: Never

        lifecycle:

          postStart:

            exec:

              command: ["/bin/sh","-c","touch /tmp/health"]

        livenessProbe:

          exec:

            command: ["test","-e","/tmp/health"]

          initialDelaySeconds: 5

          timeoutSeconds: 5

          periodSeconds: 10

        readinessProbe:

          tcpSocket:

            port: 8080

          initialDelaySeconds: 15

(7) 使上述配置文件生效,在deploy目录下执行

kubectl apply -f .

(8) 查看rocketmq 的pod情况

kubectl get pod -n wiseco (STATUS皆为RUNNING则成功启动)

(9) 查看rocketmq的service服务是否全部启动

kubectl get svc -n wiseco (如图,总共6个服务全部启动)

(10) 查看控制台状态

访问:192.168.212.176:30916 ,即可进入控制台,将ip换成自己的ip:30916可访问自己部署的控制台.进入后点击cluster,能够看到四个broker则控制台状态正常.

(11) 问题解决

如果控制台不显示各个broker,报错如下:

点击OPS,之后点击下方按钮,关闭之后点击update,看到success之后再次点击Cluster,显示正常.

至此部署完成.

参考文档: https://cloud.tencent.com/developer/article/1785729