Kubernetes持久化安装rabbitmq 3.8.34集群

藏经老曾

已于 2023-04-21 13:33:13 修改

阅读量320

点赞数

分类专栏： kubernetes 文章标签： java-rabbitmq rabbitmq kubernetes 运维容器

于 2023-04-20 10:17:50 首次发布

本文链接：https://blog.csdn.net/zhangkangren/article/details/130259956

版权

kubernetes 专栏收录该内容

6 篇文章 0 订阅

订阅专栏

部署集群已经打开promethues监控模块，已经安装了promethues的会自动监控上
所有node节点下载监控所需镜像
promethues部署
 下载rabbitmq镜像

docker pull rabbitmq:3.8.34-management

1、创建namespace

apiVersion: v1
kind: Namespace
metadata:
  labels:
    kubernetes.io/metadata.name: rabbitmq-system
  name: rabbitmq-system

2、创建ConfigMap

kind: ConfigMap
apiVersion: v1
metadata:
  name: rmq-cluster-config
  namespace: rabbitmq-system
  labels:
    addonmanager.kubernetes.io/mode: Reconcile
data:
    enabled_plugins: |
      [rabbitmq_management,rabbitmq_peer_discovery_k8s,rabbitmq_prometheus].
    rabbitmq.conf: |
      loopback_users.guest = false
      ## Clustering
      cluster_formation.peer_discovery_backend = rabbit_peer_discovery_k8s
      cluster_formation.k8s.host = kubernetes.default.svc.cluster.local
      cluster_formation.k8s.address_type = hostname
      #################################################
      # rabbitmq-system is rabbitmq-cluster's namespace#
      #################################################
      cluster_formation.k8s.hostname_suffix = .rmq-cluster.rabbitmq-system.svc.cluster.local
      cluster_formation.node_cleanup.interval = 10
      cluster_formation.node_cleanup.only_log_warning = true
      cluster_partition_handling = autoheal
      ## queue master locator
      queue_master_locator=min-masters

注释

enabled_plugins #开启的rabbitmq模块
rabbitmq_management #开启dashboard图形化管理界面
rabbitmq_peer_discovery_k8s #k8s自动发现功能
rabbitmq_prometheus #开启prometheus支持模块
loopback_users.guest #是否开启本地guest管理员账户，false 关闭，默认true
default_user #新的默认用户
default_pass #默认用户密码
cluster_formation.peer_discovery_backend #要使用Kubernetes进行对等发现，设置为k8s或Kubernetes或其模块名称rabbit_peer_discovery_k8s（注意：模块名称与插件名称略有不同）
cluster_formation.k8s.host #Kubernetes API主机名 (IP地址). 默认是使用集群服务方式kubernetes.default.svc.cluster.local
cluster_formation.k8s.address_type #当根据Kubernetes返回的pod容器列表计算对等节点列表时，可以使用主机名或IP地址。这可以使用cluster_formation.k8s.address_type键进行配置，支持的值是ip或hostname。hostname是推荐的选项，但有局限性：它只能与有状态集（也是强烈推荐的）一起使用。
cluster_formation.k8s.hostname_suffix #可以为Kubernetes返回的对等主机名添加后缀
cluster_formation.node_cleanup.interval #定期执行清理检查。默认情况下，间隔为60秒，并且可以被覆盖
cluster_formation.node_cleanup.only_log_warning #要从集群中强制删除未知节点应设置为false。
queue_master_locator #定义主队列选择策略，常用的方法min-masters选择承载最小绑定主机数量的节点,client-local选择客户机声明队列连接到的节点,random随机选择节点
cluster_partition_handling #自动处理网络分区,每种模式都有自身的优缺点，没有哪种模式是万无一失的，要根据实际情形做出相应的选择，下面简要概论以下4 个模式：ignore 模式：发生网络分区时，不做任何动作，需要人工介入；pause-minority 模式：对于对等分区的处理不够优雅，可能会关闭所有的节点。一般情况下，可应用于非跨机架、奇数节点数的集群中；pause-if-all-down 模式：对于受信节点的选择尤为考究，尤其是在集群中所有节点硬件配置相同的情况下。此种模式可以处理对等分区的情形；autoheal 模式：可以处于各个情形下的网络分区。但是如果集群中有节点处于非运行状态，则此种模式会失效；

3、创建secret

kind: Secret
apiVersion: v1
metadata:
  name: rmq-cluster-secret
  namespace: rabbitmq-system
stringData:
  cookie: erlangcookie
  password: admin
  url: amqp://admin:admin@rmq-cluster-balancer
  username: admin
type: Opaque

4、创建RBAC

apiVersion: v1
kind: ServiceAccount
metadata:
  name: rmq-cluster
  namespace: rabbitmq-system
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: rmq-cluster
  namespace: rabbitmq-system
rules:
  - apiGroups:
      - ""
    resources:
      - endpoints
    verbs:
      - get
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: rmq-cluster
  namespace: rabbitmq-system
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: rmq-cluster
subjects:
- kind: ServiceAccount
  name: rmq-cluster
  namespace: rabbitmq-system

5、创建服务

---
kind: Service
apiVersion: v1
metadata:
  labels:
    app: rmq-cluster
  name: rmq-cluster
  namespace: rabbitmq-system
spec:
  clusterIP: None
  ports:
  - name: amqp
    port: 5672
    targetPort: 5672
  selector:
    app: rmq-cluster
---
kind: Service
apiVersion: v1
metadata:
  annotations:
    prometheus.io/port: '15692'
    prometheus.io/scrape: 'true'
  labels:
    app: rmq-cluster
    type: LoadBalancer
  name: rmq-cluster-balancer
  namespace: rabbitmq-system
spec:
  ports:
  - name: http
    port: 15672
    protocol: TCP
    targetPort: 15672
  - name: amqp
    port: 5672
    protocol: TCP
    targetPort: 5672
  - name: prometheus
    port: 15692
    protocol: TCP
    targetPort: 15692
  selector:
    app: rmq-cluster
  type: NodePort

6、创建PVC

已经有了ceph的的存储，没有存储可以跳过这步，rabbitmq版本升级后需要对数据进行处理

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  annotations:
    k8s.kuboard.cn/pvcType: Dynamic
  name: rabbitmq-storage
  namespace: rabbitmq-system
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 4Gi
  storageClassName: ceph

7、创建Statefulset状态集

Stateful特点

Pod一致性：包含次序（启动、停止次序）、网络一致性。此一致性与Pod相关，与被调度到哪个node节点无关；
稳定的次序：对于N个副本的StatefulSet，每个Pod都在[0，N)的范围内分配一个数字序号，且是唯一的；
稳定的网络：Pod的hostname模式为( Statefulset 名称 ) − (statefulset名称)-(statefulset名称)−(序号)；
稳定的存储：通过VolumeClaimTemplate为每个Pod创建一个PV。删除、减少副本，不会删除相关的卷。

kind: StatefulSet
apiVersion: apps/v1
metadata:
  labels:
    app: rmq-cluster
  name: rmq-cluster
  namespace: rabbitmq-system
spec:
  replicas: 3
  selector:
    matchLabels:
      app: rmq-cluster
  serviceName: rmq-cluster
  template:
    metadata:
      labels:
        app: rmq-cluster
    spec:
      containers:
      - name: rabbitmq
        image: 192.168.17.40/rabbitmq/rabbitmq:3.8.34-management
        imagePullPolicy: IfNotPresent
        env:
        - name: RABBITMQ_DEFAULT_USER
          valueFrom:
            secretKeyRef:
              key: username
              name: rmq-cluster-secret
        - name: RABBITMQ_DEFAULT_PASS
          valueFrom:
            secretKeyRef:
              key: password
              name: rmq-cluster-secret
        - name: RABBITMQ_ERLANG_COOKIE
            valueFrom:
              secretKeyRef:
                key: cookie
                name: rmq-cluster-secret
        - name: K8S_SERVICE_NAME
          value: rmq-cluster
        - name: POD_IP
          valueFrom:
            fieldRef:
              fieldPath: status.podIP
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        - name: RABBITMQ_USE_LONGNAME
          value: "true"
        - name: RABBITMQ_NODENAME
          value: rabbit@$(POD_NAME).rmq-cluster.$(POD_NAMESPACE).svc.cluster.local
        livenessProbe:
          exec:
            command: ["rabbitmq-diagnostics", "status"]
          initialDelaySeconds: 60
          periodSeconds: 60
          timeoutSeconds: 15
        readinessProbe:
          exec:
            command: ["rabbitmq-diagnostics", "ping"]
          initialDelaySeconds: 20
          periodSeconds: 60
          timeoutSeconds: 10
        ports:
        - containerPort: 15672
          name: http
          protocol: TCP
        - containerPort: 5672
          name: amqp
          protocol: TCP
        volumeMounts:
        - mountPath: /etc/rabbitmq
          name: config-volume
#        - mountPath: /var/lib/rabbitmq/mnesia #数据持久化，没有存储可以注释掉此行和下一行
#          name: rabbitmq-storage
      serviceAccountName: rmq-cluster
      terminationGracePeriodSeconds: 30
      volumes:
      - configMap:
          items:
          - key: rabbitmq.conf
            path: rabbitmq.conf
          - key: enabled_plugins
            path: enabled_plugins
          name: rmq-cluster-config
        name: config-volume
      - name: rabbitmq-storage
          persistentVolumeClaim:
            claimName: rabbitmq-storage

# 注释
创建完成后会有3个副本
持久化到/var/lib/rabbitmq/这个目录，在开启promethues功能后访问会报错。cookie错误。

8、查看pods、pv、pvc

kubectl get pods -n rabbitmq-system
NAME            READY   STATUS    RESTARTS   AGE
rmq-cluster-0   1/1     Running   0          16m
rmq-cluster-1   1/1     Running   0          17m
rmq-cluster-2   1/1     Running   0          18m

有持久化可查看，没有则不用

# kubectl get pv -n rabbitmq-system
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                                        STORAGECLASS   REASON   AGE
pvc-4d984bea-97af-42cf-93eb-4f98afe1a5d2   4Gi        RWX            Delete           Bound    rabbitmq-system/rabbitmq-storage             ceph                    21m 


# kubectl get pvc -n rabbitmq-system
NAME               STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
rabbitmq-storage   Bound    pvc-4d984bea-97af-42cf-93eb-4f98afe1a5d2   4Gi        RWX            ceph           23m

services，此时使用的NodePort方式，可改为traefik模式

# kubectl get services -n rabbitmq-system
NAME                   TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                          AGE
rmq-cluster            ClusterIP   None            <none>        5672/TCP                         5h29m
rmq-cluster-balancer   NodePort    10.100.128.40   <none>        15672:36209/TCP,5672:31891/TCP   5h29m