Kubernetes基于helm部署Kafka_Kraft集群并取消SASL认证且开启数据持久化

Dustin.Hoffman

已于 2024-08-06 16:56:42 修改

阅读量253

点赞数

分类专栏： kubernetes 文章标签： kubernetes kafka

于 2024-08-06 16:56:18 首次发布

本文链接：https://blog.csdn.net/weixin_43334786/article/details/140958903

版权

kubernetes 专栏收录该内容

18 篇文章 7 订阅

订阅专栏

注：本文档部署Kafka时，取消了默认的SASL认证的相关配置，并开启了数据持久化。

一、添加并更新Helm仓库

helm repo add bitnami  https://charts.bitnami.com/bitnami
helm repo update bitnami

二、下载并解压kafka的Chart

helm pull bitnami/kafka
tar -xf kafka-29.3.14.tgz

三、修改values.yml

下面为修改后的示例：
我已经删除了多余的注释和默认的配置，仅保留修改后的必要的内容

image:
  # 镜像仓库，原为docker.io/bitnami/kafka:3.7.1-debian-12-r4
  registry: registry.cn-hangzhou.aliyuncs.com
  repository: zhaoll/kafka
  tag: 3.7.1-debian-12-r4
  pullPolicy: IfNotPresent
heapOpts: -Xmx1024m -Xms1024m
listeners:
  client:
    containerPort: 9092
    name: CLIENT
    protocol: PLAINTEXT  # 原为SASL_PLAINTEXT，因为我不需要访问Kafka时进行认证，所以改为PLAINTEXT
    sslClientAuth: none  # 原为""，因为我不需要访问Kafka时进行认证，所以改为none
  controller:
    name: CONTROLLER
    containerPort: 9093
    protocol: PLAINTEXT  # 同上
    sslClientAuth: none  # 同上
  interbroker:
    containerPort: 9094
    name: INTERNAL
    protocol: PLAINTEXT  # 同上
    sslClientAuth: none  # 同上
  external:
    containerPort: 9095
    name: EXTERNAL
    protocol: PLAINTEXT  # 同上
    sslClientAuth: none  # 同上
  extraListeners: []
  overrideListeners: ""
  advertisedListeners: ""
  securityProtocolMap: ""
sasl: {}  #把sasl下面的大段给注释掉或者删除，并将这里改为{}
controller:
  replicaCount: 3
  controllerOnly: false
  minId: 0
  zookeeperMigrationMode: false
  heapOpts: -Xmx1024m -Xms1024m
  persistence:
    enabled: true  # 设为true，启用持久化存储
    storageClass: "nfs-client"  # 这里配置自己的storageClass，用于自动创建PV和PVC
    accessModes:
      - ReadWriteOnce
    size: 1Gi  # 大小自己决定
    mountPath: /bitnami/kafka
  logPersistence:
    enabled: true  # 启用日志持久化
    storageClass: "nfs-client"  # 这里配置自己的storageClass，用于自动创建PV和PVC
    accessModes:
      - ReadWriteOnce
    size: 1Gi  # 大小自己决定
    mountPath: /opt/bitnami/kafka/logs

service:
  type: NodePort  #这里原为ClusterIP，我们改为NodePort
  ports:
    client: 9092
    controller: 9093
    interbroker: 9094
    external: 9095
  nodePorts:
    client: ""
    external: ""
  externalTrafficPolicy: Cluster
  headless:
    controller:
      annotations: {}
      labels: {}
    broker:
      annotations: {}
      labels: {}
kraft:
  enabled: true  # 启用Kraft，不依赖zookeeper建立集群。
  existingClusterIdSecret: ""
  clusterId: ""
  controllerQuorumVoters: ""

四、开始安装

[root@master1 kafka]# helm install kafka -f values.yaml bitnami/kafka

大致解释一下上面输出的内容：

[root@master1 kafka]# helm install kafka -f values.yaml bitnami/kafka
# 安装信息，主要是kafka的版本和Chart版本
NAME: kafka
LAST DEPLOYED: Tue Aug  6 16:30:13 2024
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
CHART NAME: kafka
CHART VERSION: 29.3.14
APP VERSION: 3.7.1

** Please be patient while the chart is being deployed **

# 可以通过kafka.default.svc.cluster.local名称加上9092端口在k8s集群内部访问访问Kafka集群
Kafka can be accessed by consumers via port 9092 on the following DNS name from within your cluster:

    kafka.default.svc.cluster.local

# 生产者可以通过下面三个节点名称来访问Kafka
Each Kafka broker can be accessed by producers via port 9092 on the following DNS name(s) from within your cluster:

    kafka-controller-0.kafka-controller-headless.default.svc.cluster.local:9092
    kafka-controller-1.kafka-controller-headless.default.svc.cluster.local:9092
    kafka-controller-2.kafka-controller-headless.default.svc.cluster.local:9092

# 创建一个pod用于访问Kafka
To create a pod that you can use as a Kafka client run the following commands:
    # 这是创建pod的命令
    kubectl run kafka-client --restart='Never' --image registry.cn-hangzhou.aliyuncs.com/zhaoll/kafka:3.7.1-debian-12-r4 --namespace default --command -- sleep infinity
    # 这是进入pod内部
    kubectl exec --tty -i kafka-client --namespace default -- bash

    PRODUCER:
    # 在Pod里执行下面这行命令，开启一个生产者，用于发送消息
        kafka-console-producer.sh \
            --broker-list kafka-controller-0.kafka-controller-headless.default.svc.cluster.local:9092,kafka-controller-1.kafka-controller-headless.default.svc.cluster.local:9092,kafka-controller-2.kafka-controller-headless.default.svc.cluster.local:9092 \
            --topic test

    CONSUMER:
    # 在Pod里执行下面这行命令，开启一个消费者，用于接收消息
        kafka-console-consumer.sh \
            --bootstrap-server kafka.default.svc.cluster.local:9092 \
            --topic test \
            --from-beginning
# 默认镜像被替换了，会在此出现警告
Substituted images detected:
  - registry.cn-hangzhou.aliyuncs.com/zhaoll/kafka:3.7.1-debian-12-r4

按照上面的提示，创建一个测试用的Pod：

kubectl run kafka-client --restart='Never' --image registry.cn-hangzhou.aliyuncs.com/zhaoll/kafka:3.7.1-debian-12-r4 --namespace default --command -- sleep infinity

安装完成后，查看Pod和SVC：

[root@master1 kafka]# kubectl get pod,svc
NAME                                          READY   STATUS    RESTARTS         AGE
pod/kafka-client                              1/1     Running   0                3m
pod/kafka-controller-0                        1/1     Running   0                4m
pod/kafka-controller-1                        1/1     Running   0                4m
pod/kafka-controller-2                        1/1     Running   0                4m
pod/nfs-client-provisioner-6f5897fd65-28qlw   1/1     Running   0                15d

NAME                                TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)                      AGE
service/kafka                       NodePort    10.108.113.8   <none>        9092:32476/TCP               4m
service/kafka-controller-headless   ClusterIP   None           <none>        9094/TCP,9092/TCP,9093/TCP   4m
service/kubernetes                  ClusterIP   10.96.0.1      <none>        443/TCP                      18d

六、创建生产者和消费者进行测试

生产者：

[root@master1 kafka]# kubectl exec --tty -i kafka-client --namespace default -- bash
I have no name!@kafka-client:/$ kafka-console-producer.sh \
>             --broker-list kafka-controller-0.kafka-controller-headless.default.svc.cluster.local:9092,kafka-controller-1.kafka-controller-headless.default.svc.cluster.local:9092,kafka-controller-2.kafka-controller-headless.default.svc.cluster.local:9092 \
>             --topic test
>我超级牛逼的
>

消费者：

[root@master1 kafka]# kubectl exec --tty -i kafka-client --namespace default -- bash
I have no name!@kafka-client:/$ kafka-console-consumer.sh \
>             --bootstrap-server kafka.default.svc.cluster.local:9092 \
>             --topic test \
>             --from-beginning
我超级牛逼的