k8s 部署rocketmq集群

@土豆

已于 2023-04-26 20:17:50 修改

阅读量7.5k

点赞数 28

文章标签： kubernetes java-rocketmq rocketmq

于 2022-10-25 20:53:31 首次发布

本文链接：https://blog.csdn.net/weixin_44770684/article/details/127520224

版权

一、前言

二、构建 rockermq镜像

三、构建rocketmq-dashboard镜像

四、rocketmq部署文件

五、rocketmq-dashboard部署文件

六、调整副本数案例

七、附加信息

八、附录

一、前言

由于网上找到的大部分部署方案以及rocketmq-operator都是不能很灵活的调整副本数，毕竟每个副本对应其唯一的配置的文件，不同的broker实例使用的配置文件都是不一样的，都存在差异，一旦副本变化了，就不能很好的和broker 的配置文件内容一一对应起来。

在这里创建一个基于k8s部署单master以及多master部署rocketmq集群，并且只需要一个broker配置文件，多个broker实例会自动基于该broker配置文件模板，自动生成不同broker实例的broker配置文件，扩容或者伸缩rocketmq集群 nameserver或者broker副本数的时候不需要理会配置文件，仅仅是调整实例的副本即可。

备注：该方案不适合有slave节点的部署方式。

二、构建 rockermq镜像

rocketmq-namesrv和 rocketmq-broker共用同一个镜像，仅仅是启动命令和启动参数不一样，后期可灵活的通过调整启动命令和启动参数来实现不同的效果（比如通过挂载configMap的方式自定义rocketmq的配置文件，而不需要重建rocketmq的镜像。

## docker build -t test/rocketmq:v1 . --no-cache

### 构建rocketmq 镜像
##  cat Dockerfile

FROM   docker.io/library/openjdk:8u102-jdk AS JDK

LABEL "作者"=fanqietudou  mail=123456@qq.com

RUN     rm -vf /etc/localtime \
     && ln -s /usr/share/zoneinfo/Asia/Shanghai /etc/localtime \
     && echo "Asia/Shanghai" > /etc/timezone \
     && export LANG=zh_CN.UTF-8

RUN     curl -k https://mirrors.tuna.tsinghua.edu.cn/apache/rocketmq/4.9.4/rocketmq-all-4.9.4-bin-release.zip \
             -o /tmp/rocketmq-all-4.9.4-bin-release.zip    \
     && unzip /tmp/rocketmq-all-4.9.4-bin-release.zip -d /tmp/ \
     && mv /tmp/rocketmq-all-4.9.4-bin-release /opt/rocketmq \
     && rm -rf /tmp/*

RUN     sed -ir '/-Xmx/c JAVA_OPT=${JAVA_OPT}' /opt/rocketmq/bin/runserver.sh \
     && sed -ir '/-Xmx/c JAVA_OPT=${JAVA_OPT}' /opt/rocketmq/bin/runbroker.sh

##  运行 MQ 应用时候可以通过环境变量设置 jvm 数值，如：JAVA_OPT="-server -Xms2g -Xmx2g -XX:MetaspaceSize=128m -XX:MaxMetaspaceSize=320m"

ENV     ROCKETMQ_HOME=/opt/rocketmq

WORKDIR $ROCKETMQ_HOME

三、构建rocketmq-dashboard镜像

rocketmq-dashboard是一个可视化的rocketmq集群运维监控工具。

# cat Dockerfile
# docker build -t test/rocketmq-dashboard:v1 . --no-cache

# docker run -d --name rocketmq-dashboard -e "JAVA_OPTS=-Drocketmq.namesrv.addr=127.0.0.1:9876" -p 8080:8080 -t apacherocketmq/rocketmq-dashboard:latest
#
# 下载地址 https://mirrors.tuna.tsinghua.edu.cn/apache/rocketmq/rocketmq-dashboard/1.0.0/rocketmq-dashboard-1.0.0-source-release.zip

# github 地址 https://github.com/apache/rocketmq-dashboard.git
#  git clone https://github.com/apache/rocketmq-dashboard.git

# 下载源码后，通过mvn 编译 得到 rocketmq-dashboard.jar
# 编译命令  mvn clean package -Dmaven.test.skip=true docker:build

# 也可以直接 拉取别人做好的镜像 docker pull apacherocketmq/rocketmq-dashboard:latest
#  也可以直接使用编译后，代码生成的 Dockerfile 文件构建 镜像
#  由于我的环境 无法直接拉取 java:8 基础镜像，所以我 换了一个基础镜像

# FROM java:8

FROM   docker.io/library/openjdk:8u102-jdk AS JDK
VOLUME /tmp
ADD rocketmq-dashboard-*.jar rocketmq-dashboard.jar
RUN sh -c 'touch /rocketmq-dashboard.jar'
ENV JAVA_OPTS=""
ENTRYPOINT [ "sh", "-c", "java $JAVA_OPTS -jar /rocketmq-dashboard.jar" ]

四、rocketmq部署文件

该部署文件包含了rocketmq-nameserver和rocketmq-broker两部分内容，此部分要重点关注各自的启动命令和启动参数，启动命令和启动参数（command）是部署成功的关键，不同的需求，需要适当调整启动命令和启动参数（这里我只做简单的启动参数做例子）。

## cat rocketmq.yaml 

## kubectl apply -f rocketmq.yaml
---
# rocketmq-namesrv 部署文件

apiVersion: apps/v1
kind: StatefulSet
metadata:
  annotations:
  generation: 2
  labels:
    app: rocketmq-namesrv
  name: rocketmq-namesrv
#  namespace: mq
spec:
  podManagementPolicy: OrderedReady
  replicas: 2
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: rocketmq-namesrv
  serviceName: rocketmq-namesrv
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: rocketmq-namesrv
    spec:
      containers:
      - command:
        - bin/mqnamesrv
        env:
        - name: JAVA_OPT
          value: -server -Xms2g -Xmx2g -XX:MetaspaceSize=128m -XX:MaxMetaspaceSize=320m
        image: test/rocketmq:v1
        imagePullPolicy: IfNotPresent
        name: rocketmq-namesrv
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30
  updateStrategy:
    rollingUpdate:
      partition: 0
    type: RollingUpdate
---

# rocketmq-broker 部署文件

apiVersion: apps/v1
kind: StatefulSet
metadata:
  annotations:
  labels:
    app: rocketmq-broker
  name: rocketmq-broker
#  namespace: mq
spec:
  podManagementPolicy: OrderedReady
  replicas: 2
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: rocketmq-broker
  serviceName: rocketmq-broker
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: rocketmq-broker
    spec:
      containers:
      - command:
        - bin/mqbroker
        - --namesrvAddr=rocketmq-namesrv:9876
        env:
        - name: JAVA_OPT
          value: -server -Xms4g -Xmx4g
        image: test/rocketmq:v1
        imagePullPolicy: IfNotPresent
        name: rocketmq-broker
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30
  updateStrategy:
    rollingUpdate:
      partition: 0
    type: RollingUpdate
---

# rocketmq-namesrv 服务文件

apiVersion: v1
kind: Service
metadata:
  annotations:
  labels:
    app: rocketmq-namesrv
  name: rocketmq-namesrv
#  namespace: mq
spec:
  clusterIP: None
  ports:
  - port: 9876
    protocol: TCP
    targetPort: 9876
  selector:
    app: rocketmq-namesrv
  sessionAffinity: None
  type: ClusterIP
---

五、rocketmq-dashboard部署文件

前四个步骤已经完成创建rocketmq集群，接下来是部署一个能实现运维监控rocketmq的可视化web应用。

部署rocketmq-dashboard应用时候重点关注部署文件里面的env环境变量参数JAVA_OPTS，该env环境变量（JAVA_OPTS）决定了应用是否能成功连接到 rocketmq-namesrv 服务。

---
# rocketmq-dashboard 部署文件

# cat rocketmq-dashboard.yaml

# kubectl apply -f rocketmq-dashboard.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
  labels:
    app: rocketmq-dashboard
  name: rocketmq-dashboard
#  namespace: mq
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: rocketmq-dashboard
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: rocketmq-dashboard
    spec:
      containers:
      - env:
        - name: JAVA_OPTS
          value: -Drocketmq.namesrv.addr=rocketmq-namesrv:9876 -Dcom.rocketmq.sendMessageWithVIPChannel=false
        image: test/rocketmq-dashboard:v1
        imagePullPolicy: IfNotPresent
        name: rocketmq-dashboard
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30
---

# rocketmq-dashboard 服务文件

apiVersion: v1
kind: Service
metadata:
  annotations:
  labels:
    app: rocketmq-dashboard
  name: rocketmq-dashboard
#  namespace: mq
spec:
  clusterIP: 10.96.149.156
  ports:
  - port: 8080
    protocol: TCP
    targetPort: 8080
  selector:
    app: rocketmq-dashboard
  sessionAffinity: None
  type: ClusterIP
---

六、调整副本数案例

## 副本数按照自己需求调整即可

## 1、 调整 rocketmq-namesrv 副本数

kubectl scale sts rocketmq-namesrv --replicas=3


## 2、调整 rocketmq-broker 副本数

kubectl scale sts rocketmq-broker --replicas=3

七、附加信息

由于我的环境是使用Ingress方式访问，在这里我就不在写ingress配置了，如果有需要可以将上边的 rocketmq-dashboard得服务 service修改成nodeport形式访问。

rocketmq-dashboard应用默认端口是8080，登录账号和密码都是: admin。

默认情况rocketmq-dashboard的访问地址是： http://ip:8080 ，在k8s环境，rocketmq-dashboard的访问端口取决于nodeport映射的实际端口。

八、附录

该部分内容是我们测试环境完整的部署文件，该部署方法是挂载自定义的可通过configMap 修改的配置文件。（rocketmq-dashboard是通过Ingress提供外网访问的，该Ingress部署内容部分就不展示了）

### cat rocketmq.yaml
### kubectl apply -f rocketmq.yaml


---
###########################################################

### 1、部署 rocketmq-namesrv

###########################################################
---
### rocketmq-namesrv configMap 配置文件 (无配置)
### rocketmq-namesrv 没有必要挂载 配置文件使用 默认的就好了

---
###  rocketmq-namesrv 部署文件，已挂载 configMap 配置文件

apiVersion: apps/v1
kind: StatefulSet
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
  labels:
    app: rocketmq-namesrv
  name: rocketmq-namesrv
  # namespace: mq
spec:
  podManagementPolicy: OrderedReady
  replicas: 2
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: rocketmq-namesrv
  serviceName: rocketmq-namesrv
  template:
    metadata:
      annotations:
        kubectl.kubernetes.io/restartedAt: "2022-10-26T09:27:22+08:00"
      creationTimestamp: null
      labels:
        app: rocketmq-namesrv
    spec:
      containers:
      - command:
        - bin/mqnamesrv
        image: 172.18.6.97/test/rocketmq:v1
        imagePullPolicy: IfNotPresent
        name: rocketmq
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30
  updateStrategy:
    rollingUpdate:
      partition: 0
    type: RollingUpdate
---

### rocketmq-namesrv 服务文件

apiVersion: v1
kind: Service
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
  labels:
    app: rocketmq-namesrv
  name: rocketmq-namesrv
  # namespace: mq
spec:
  clusterIP: None
  ports:
  - port: 9876
    protocol: TCP
    targetPort: 9876
  selector:
    app: rocketmq-namesrv
  sessionAffinity: None
  type: ClusterIP

---

###########################################################

### 2、部署 rocketmq-broker

###########################################################

---
### rocketmq-broker configMap 配置文件
apiVersion: v1
data:
  broker.conf: |
    # 所属集群名字(同一主从下: Master 和 Slave 名称要一致)
     brokerClusterName = RocketMQ-cluster

     # Broker 名字，注意此处不同的配置文件填写的不一样
     #brokerName = broker-a

     # 0 表示 Master，> 0 表示 Slave
     brokerId = 0

     # Broker 对外服务的监听端口, 如果一台机器上启动了多个Broker，则要设置不同的端口号，避免冲突
     listenPort = 10911

     # nameServer地址，如果nameServer是多台集群的话，就用分号分割，比如
     # namesrvAddr=192.168.0.1:9876;192.168.0.2:9876

     # 是否允许 Broker 自动创建Topic，建议线下开启，线上关闭
     autoCreateTopicEnable = true

     # 是否允许 Broker 自动创建订阅组，建议线下开启，线上关闭
     autoCreateSubscriptionGroup = true

     # 与fileReservedTime参数呼应，表明在几点做消息删除动作，默认值04表示凌晨4点
     deleteWhen = 04

     # 在磁盘上保存消息的时长，单位是小时，自动删除超时的消息
     fileReservedTime = 48

     # brokerRole有3种：SYNC_MASTER、ASYNC_MASTER、SLAVE
     # 关键词 SYNC 和 ASYNC 表示 Master 和 Slave 之间同步消息的机制
     # SYNC 的意思是当 Slave 和 Master 消息同步完成后，再返回发送成功的状态
     brokerRole = ASYNC_MASTER

     # 刷盘方式 ASYNC_FLUSH 异步刷盘; SYNC_FLUSH 同步刷盘
     flushDiskType = ASYNC_FLUSH

    # #存储路径
    # storePathRootDir = /opt/rocketmq/store
    # #commitLog 存储路径
    # storePathCommitLog = /opt/rocketmq/store/commitlog
    # #消费队列存储路径存储路径
    # storePathConsumeQueue = /opt/rocketmq/store/consumequeue
    # #消息索引存储路径
    # storePathIndex=/opt/rocketmq/store/index
    # #checkpoint 文件存储路径
    # storeCheckpoint=/opt/rocketmq/store/checkpoint

     #限制的消息大小
     maxMessageSize=65536
kind: ConfigMap
metadata:
  name: rocketmq-broker
  # namespace: mq

---
### rocketmq-broker 部署文件 ，已挂载 configMap 作为配置文件

apiVersion: apps/v1
kind: StatefulSet
metadata:
  annotations:
  labels:
    app: rocketmq-broker
  name: rocketmq-broker
  # namespace: mq
spec:
  podManagementPolicy: OrderedReady
  replicas: 2
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: rocketmq-broker
  serviceName: rocketmq-broker
  template:
    metadata:
      annotations:
        cattle.io/timestamp: "2022-10-26T01:46:48Z"
      creationTimestamp: null
      labels:
        app: rocketmq-broker
    spec:
      affinity: {}
      containers:
      - command:
        - bin/mqbroker
        - --configFile=conf/broker.conf
        - --namesrvAddr=rocketmq-namesrv:9876
        image: 172.18.6.97/test/rocketmq:v1
        imagePullPolicy: IfNotPresent
        name: rocketmq
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /opt/rocketmq/conf/broker.conf
          name: rocketmq-broker-conf
          readOnly: true
          subPath: broker.conf
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30
      volumes:
      - configMap:
          defaultMode: 420
          name: rocketmq-broker
        name: rocketmq-broker-conf
  updateStrategy:
    rollingUpdate:
      partition: 0
    type: RollingUpdate

---

###########################################################

### 3、部署 rocketmq-dashboard

###########################################################
---
### rocketmq-dashboard configMap 文件

apiVersion: v1
data:
  users.properties: |2-

    # This file supports hot change, any change will be auto-reloaded without Dashboard restarting.
    # Format: a user per line, username=password[,N] #N is optional, 0 (Normal User); 1 (Admin)

    # Define Admin
    admin=admin,1

    # Define Users
    user1=user1
    user2=user2
kind: ConfigMap
metadata:
  name: rocketmq-dashboard
  # namespace: mq

---

### rocketmq-dashboard 部署文件

apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
  labels:
    app: rocketmq-dashboard
  name: rocketmq-dashboard
  # namespace: mq
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: rocketmq-dashboard
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: rocketmq-dashboard
    spec:
      affinity: {}
      containers:
      - env:
        - name: JAVA_OPTS
          value: -Drocketmq.namesrv.addr=rocketmq-namesrv:9876 -Dcom.rocketmq.sendMessageWithVIPChannel=false
        image: 172.18.6.97/test/rocketmq-dashboard:v1
        imagePullPolicy: IfNotPresent
        name: rocketmq-dashboard
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: tmp/rocketmq-console/data/users.properties
          name: users-properties
          readOnly: true
          subPath: users.properties
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30
      volumes:
      - configMap:
          defaultMode: 420
          name: rocketmq-dashboard
        name: users-properties

---

### rocketmq-dashboard  服务文件

apiVersion: v1
kind: Service
metadata:
  annotations:
  labels:
    app: rocketmq-dashboard
  name: rocketmq-dashboard
  # namespace: mq
spec:
  clusterIP: 10.96.149.156
  ports:
  - port: 8080
    protocol: TCP
    targetPort: 8080
  selector:
    app: rocketmq-dashboard
  sessionAffinity: None
  type: ClusterIP

---