在K8s上搭建Hazelcast 集群

淹不死的鱼丶

于 2024-08-26 12:01:10 发布

阅读量935

点赞数 24

分类专栏：组件服务文章标签： kubernetes

本文链接：https://blog.csdn.net/p73133_55/article/details/141559430

版权

组件服务专栏收录该内容

1 篇文章 0 订阅

订阅专栏

Hazelcast基于k8s的集群搭建

背景

最近在研究Seatunnel的源码，Seatunnel是基于hazelcast实现的去中心化实现，同时hazelcast作为基于网络网格的分布式缓存服务，同时有一个Python项目需要做性能优化，想研究下，hazelcast是否可以在这个项目上进行使用，开发部署都是基于k8s的，所以，打算基于k8s搭建一个hazelcast集群。

搭建方式（挺简单的）

根据官方文档，通过k8s部署hazelcast集群，基本分为两步，依次执行：

初始化rabc.yaml文件，这里主要处理hazelcast启动集群所需的角色和权限，同时要和使用的hazelcast对应，大家可以直接执行以下命令，也可将rabc.yaml文件下载到本地直接apply或者create

kubectl apply -f https://raw.githubusercontent.com/hazelcast/hazelcast/master/kubernetes-rbac.yaml

rabc.yaml 文件内容如下，可以直接copy使用,subjects.kind.namespace 可以根据自己需要进行调整，我的k8s中使用的namespace是hazelcast，如果不设置，就需要进入到自己使用的namespace进行/create/apply操作

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: hazelcast-cluster-role
rules:
  - apiGroups:
      - ""
      # Access to apps API is only required to support automatic cluster state management
      # when persistence (hot-restart) is enabled.
      - apps
    resources:
      - endpoints
      - pods
      - nodes
      - services
      # Access to statefulsets resource is only required to support automatic cluster state management
      # when persistence (hot-restart) is enabled.
      - statefulsets
    verbs:
      - get
      - list
      # Watching resources is only required to support automatic cluster state management
      # when persistence (hot-restart) is enabled.
      - watch
  - apiGroups:
      - "discovery.k8s.io"
    resources:
      - endpointslices
    verbs:
      - get
      - list

---

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: hazelcast-cluster-role-binding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: hazelcast-cluster-role
subjects:
  - kind: ServiceAccount
    name: default
    namespace: hazelcast

配置启动Hazelcast集群的yaml文件,这里配置了三个服务，一个是hazelcast集群启动配置，启动statefulset模式，二个是集群启动hazelcast服务自身配置通过configmap支持，三一个就是hazelcast集群暴露服务的方式，因为我这边需要在k8s外访问，所以只能支持NodePort或者LoadBalancer,可以直接使用如下yaml文件，修改namespace和镜像地址即可

#hazelcast集群配置
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: hazelcast
  namespace: hazelcast
  labels:
    role: hazelcast
spec:
  serviceName: hazelcast
  replicas: 3
  selector:
    matchLabels:
      app: hazelcast
  template:
    metadata:
      labels:
        app: hazelcast
    spec:
      volumes: # Pod模板的卷配置开始
        - name: hazelcast-config
          configMap:
            name: hazelcast-config
      containers:
      - name: hazelcast
        # Hazelcast 5.5.0 image with Kubernetes discovery enabled，如果需要科学上网，可以先放到本地镜像仓库
        image: 127.0.0.1/public/hazelcast/hazelcast:5.5.0-slim-jdk21
        ports:
        - containerPort: 5701
        resources:
          requests:
            memory: "512Mi"
            cpu: "500m"
          limits:
            memory: "1Gi"
            cpu: "1"
        env:
        - name: JAVA_OPTS
          value: "-Dhazelcast.config=/hazelcast/hazelcast.yaml"
        - name: HZ_CLUSTER_NAME
          value: "seatunnel"
        - name: HZ_KUBERNETES_ENABLED
          value: "true"
        - name: HZ_KUBERNETES_SERVICE_DNS
          value: "hazelcast.hazelcast.svc.cluster.local"
        - name: HZ_KUBERNETES_NAMESPACE
          value: "hazelcast"
        - name: HZ_KUBERNETES_USE_NODE_NAME_AS_EXTERNAL_ADDRESS
          value: "true"
        - name: HZ_KUBERNETES_NODE_NAME
          valueFrom:
            fieldRef:
              fieldPath: spec.nodeName
        volumeMounts:
          - name: hazelcast-config
            mountPath: /hazelcast


---

apiVersion: v1
kind: ConfigMap
metadata:
  name: hazelcast-config
  namespace: hazelcast
data:
  hazelcast.yaml: |
    hazelcast:
      cluster-name: seatunnel
      network:
        join:
          multicast:
            enabled: false
          kubernetes:
            enabled: true
            namespace: hazelcast
            service-name: hazelcast
              
              

---

apiVersion: v1
kind: Service
metadata:
  name: hazelcast
  namespace: hazelcast
  labels:
    role: hazelcast
spec:
  type: NodePort
  ports:
  - port: 5701
    targetPort: 5701
    nodePort: 30071
  selector:
    app: hazelcast

yaml文件整合

可以将rabc.yaml和hazelcast-cluster.yaml文件整合成一个yaml文件，这样更方便维护和管理，整合后的yaml文件如下，想必分步执行，整合后，只需要执行一个文件即可完成搭建

#初始化RABC
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: hazelcast-cluster-role
  namespace: hazelcast
rules:
  - apiGroups:
      - ""
      # Access to apps API is only required to support automatic cluster state management
      # when persistence (hot-restart) is enabled.
      - apps
    resources:
      - endpoints
      - pods
      - nodes
      - services
      # Access to statefulsets resource is only required to support automatic cluster state management
      # when persistence (hot-restart) is enabled.
      - statefulsets
    verbs:
      - get
      - list
      # Watching resources is only required to support automatic cluster state management
      # when persistence (hot-restart) is enabled.
      - watch
  - apiGroups:
      - "discovery.k8s.io"
    resources:
      - endpointslices
    verbs:
      - get
      - list

---

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: hazelcast-cluster-role-binding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: hazelcast-cluster-role
subjects:
  - kind: ServiceAccount
    name: default
    #这里我的k8s中使用的namespace是hazelcast
    namespace: hazelcast

---
#hazelcast集群配置
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: hazelcast
  namespace: hazelcast
  labels:
    role: hazelcast
spec:
  serviceName: hazelcast
  replicas: 3
  selector:
    matchLabels:
      app: hazelcast
  template:
    metadata:
      labels:
        app: hazelcast
    spec:
      volumes: # Pod模板的卷配置开始
        - name: hazelcast-config
          configMap:
            name: hazelcast-config
      containers:
      - name: hazelcast
        # Hazelcast 5.5.0 image with Kubernetes discovery enabled，如果需要科学上网，可以先放到本地镜像仓库（hazelcast/hazelcast:5.5.0-slim-jdk21）
        image: 127.0.0.1/public/hazelcast/hazelcast:5.5.0-slim-jdk21
        ports:
        - containerPort: 5701
        resources:
          requests:
            memory: "512Mi"
            cpu: "500m"
          limits:
            memory: "1Gi"
            cpu: "1"
        env:
        - name: JAVA_OPTS
          value: "-Dhazelcast.config=/hazelcast/hazelcast.yaml"
        - name: HZ_CLUSTER_NAME
          value: "seatunnel"
        - name: HZ_KUBERNETES_ENABLED
          value: "true"
        - name: HZ_KUBERNETES_SERVICE_DNS
          value: "hazelcast.hazelcast.svc.cluster.local"
        - name: HZ_KUBERNETES_NAMESPACE
          value: "hazelcast"
        - name: HZ_KUBERNETES_USE_NODE_NAME_AS_EXTERNAL_ADDRESS
          value: "true"
        - name: HZ_KUBERNETES_NODE_NAME
          valueFrom:
            fieldRef:
              fieldPath: spec.nodeName
        volumeMounts:
          - name: hazelcast-config
            mountPath: /hazelcast


---
#hazelcast客户端连接配置文件，通过configmap方式提供
apiVersion: v1
kind: ConfigMap
metadata:
  name: hazelcast-config
  namespace: hazelcast
data:
  hazelcast.yaml: |
    hazelcast:
      cluster-name: seatunnel
      network:
        join:
          multicast:
            enabled: false
          kubernetes:
            enabled: true
            namespace: hazelcast
            service-name: hazelcast
              
              

---
#暴露服务的Service
apiVersion: v1
kind: Service
metadata:
  name: hazelcast
  namespace: hazelcast
  labels:
    role: hazelcast
spec:
  type: NodePort
  ports:
  - port: 5701
    targetPort: 5701
    nodePort: 30071
  selector:
    app: hazelcast

本地测试

smartRouting 必须设置false，因为本地和k8s pod无法直接通信

Hazelcast JAVA Client:
可以直接在代码中配置hazelcast，也可以通过配置文件，这里使用xml文件,address使用的k8s的node节点和服务在节点暴露的端口

<hazelcast-client xmlns="http://www.hazelcast.com/schema/client-config"
                  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
                  xsi:schemaLocation="http://www.hazelcast.com/schema/client-config
                  http://www.hazelcast.com/schema/client-config/hazelcast-client-config-4.0.xsd">
    <cluster-name>seatunnel</cluster-name>
    <network>
        <cluster-members>
            <address>10.122.0.0:30071</address>
            <address>10.122.0.1:30071</address>
            <address>10.122.0.2:30071</address>

        </cluster-members>
    </network>
</hazelcast-client>

POM依赖

    <dependencies>
        <dependency>
            <groupId>com.hazelcast</groupId>
            <artifactId>hazelcast</artifactId>
            <version>5.1</version>
        </dependency>
    </dependencies>

JAVA 样例代码：

package org.apache.seatunnel;

import com.hazelcast.client.HazelcastClient;
import com.hazelcast.client.config.ClientConfig;
import com.hazelcast.client.config.XmlClientConfigBuilder;
import com.hazelcast.collection.IQueue;
import com.hazelcast.core.HazelcastInstance;

import java.io.IOException;

public class HazelcastClientDemoConsumer {
    public static void main(String[] args) throws IOException {
    // 此处可以使用绝对路径或者相对路径
        ClientConfig config = new XmlClientConfigBuilder("classpath:hazelcast-client.xml").build();
        config.getNetworkConfig().setSmartRouting(false);
        HazelcastInstance hzClient = HazelcastClient.newHazelcastClient(config);
        // Read from map
        System.out.println("hzClient = " + hzClient);
        IQueue<String> queue = hzClient.getQueue("seatunnel_job");

        for (int i = 0; i < 50; i++) {
            queue.add("item 1" + i);
        }


        for (int i = 0; i < 50; i++) {
            System.out.println(queue.poll());
        }

    }
}

Python 验证简单代码

import logging
import hazelcast logging.basicConfig(level=logging.INFO) 
client = hazelcast.HazelcastClient(
cluster_members=["<EXTERNAL-IP>"],
smart_routing=False
)