kafka集群 kubernetes_使用Kubernetes部署Kafka集群

Apache Kafka是一种流行的分布式流式消息平台。Kafka生产者将数据写入分区主题，这些主题通过可配置的副本存储到broker群集上。 消费者来消费存储在broker的分区生成的数据。

注意：详细信息可以在这里找到。 可以在此处了解有关在Kubernetes上运行Kafka群集的更多信息。

apiVersion: policy/v1beta1

kind: PodDisruptionBudget

metadata:

name: kafka-pdb

spec:

selector:

matchLabels:

app: kafka

maxUnavailable: 1

---apiVersion: apps/v1beta1

kind: StatefulSet

metadata:

name: kafka

spec:

serviceName: kafka-hs

replicas: 3

podManagementPolicy: Parallel

updateStrategy:

type: RollingUpdate

template:

metadata:

labels:

app: kafka

spec:

affinity:

podAntiAffinity:

requiredDuringSchedulingIgnoredDuringExecution:

-labelSelector: matchExpressions:

-key: "app" operator: In

values:

-kafka topologyKey: "kubernetes.io/hostname"

podAffinity:

preferredDuringSchedulingIgnoredDuringExecution:

-weight: 1 podAffinityTerm:

labelSelector:

matchExpressions:

-key: "app" operator: In

values:

-zk topologyKey: "kubernetes.io/hostname"

terminationGracePeriodSeconds: 300

containers:

-name: k8skafka imagePullPolicy: Always

image: gcr.io/google_containers/kubernetes-kafka:1.0-10.2.1

resources:

requests:

memory: "1Gi"

cpu: "0.5"

ports:

-containerPort: 9093 name: server

command:

-sh --c -"exec kafka-server-start.sh /opt/kafka/config/server.properties --override broker.id=${HOSTNAME##*-} \ --override listeners=PLAINTEXT://:9093 \ --override zookeeper.connect=zk-cs.default.svc.cluster.local:2181 \ --override log.dir=/var/lib/kafka \ --override auto.create.topics.enable=true \ --override auto.leader.rebalance.enable=true \ --override background.threads=10 \ --override compression.type=producer \ --override delete.topic.enable=false \ --override leader.imbalance.check.interval.seconds=300 \ --override leader.imbalance.per.broker.percentage=10 \ --override log.flush.interval.messages=9223372036854775807 \ --override log.flush.offset.checkpoint.interval.ms=60000 \ --override log.flush.scheduler.interval.ms=9223372036854775807 \ --override log.retention.bytes=-1 \ --override log.retention.hours=168 \ --override log.roll.hours=168 \ --override log.roll.jitter.hours=0 \ --override log.segment.bytes=1073741824 \ --override log.segment.delete.delay.ms=60000 \ --override message.max.bytes=1000012 \ --override min.insync.replicas=1 \ --override num.io.threads=8 \ --override num.network.threads=3 \ --override num.recovery.threads.per.data.dir=1 \ --override num.replica.fetchers=1 \ --override offset.metadata.max.bytes=4096 \ --override offsets.commit.required.acks=-1 \ --override offsets.commit.timeout.ms=5000 \ --override offsets.load.buffer.size=5242880 \ --override offsets.retention.check.interval.ms=600000 \ --override offsets.retention.minutes=1440 \ --override offsets.topic.compression.codec=0 \ --override offsets.topic.num.partitions=50 \ --override offsets.topic.replication.factor=3 \ --override offsets.topic.segment.