05 部署zookeeper和kafka集群

104 篇文章 133 订阅

部署zookeeper和kafka集群

文章目录

提供者:MappleZF

版本:1.0.0

一、部署zookeeper

1.1 准备环境

准备安装包、安装jdk

1.2 安装解压

tar xf apache-zookeeper-3.6.2-bin.tar.gz -C /opt/
cd /opt && mv apache-zookeeper-3.6.2-bin/ zookeeper
mkdir /opt/zookeeper/{data,logs}

1.3 添加DNS域名解析或修改hosts文件

[root@lb03.host.com:/root]# vim /var/named/lowan.com.zone
zk01            A       192.168.13.101
zk02            A       192.168.13.102
zk03            A       192.168.13.103


1.4 配置zoo.cfg文件

cd /opt/zookeeper/conf/
cp -p zoo_sample.cfg zoo.cfg


conf]# vim zoo.cfg
内容如下:
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/opt/zookeeper/data
dataLogDir=/opt/zookeeper/logs
clientPort=2181
server.1=192.168.13.101:2888:3888
server.2=192.168.13.102:2888:3888
server.3=192.168.13.103:2888:3888
autopurge.purgeInterval=12
max_session_timeout=40000
min_session_timeout=4000

分发cfg文件
scp zoo.cfg k8smaster02:/opt/zookeeper/conf/
scp zoo.cfg k8smaster03:/opt/zookeeper/conf/

1.5 创建myid文件

# 每个服务器上的myid内容都不同,且需要保证和自己的zoo.cfg配置文件中"server.id=host:port:port"的id值一致
# id的范围是1~255

[root@k8smaster01.host.com:/opt/zookeeper/conf]# echo "1" > /opt/zookeeper/data/myid
[root@k8smaster02.host.com:/opt]# echo "2" > /opt/zookeeper/data/myid
[root@k8smaster03.host.com:/opt]# echo "3" > /opt/zookeeper/data/myid

1.6 启动zookeeper

/opt/zookeeper/bin/zkServer.sh start
ps -ef | grep zookeeper

1.7 添加守护进程

vim /etc/systemd/system/zookeeper.service

[Unit]
Description=zookeeper server
Requires=network.target remote-fs.target
After=network.target remote-fs.target

[Service]
Type=forking
User=root
Environment=JAVA_HOME=/usr/local/java
ExecStart=/bin/sh -c '/opt/zookeeper/bin/zkServer.sh start /opt/zookeeper/conf/zoo.cfg'
ExecStop=/opt/zookeeper/bin/zkServer.sh stop
Restart=on-failure
RestartSec=2s
[Install]
WantedBy=multi-user.target

:x 保存退出
/opt/zookeeper/bin/zkServer.sh stop

systemctl daemon-reload
systemctl enable zookeeper.service
systemctl start zookeeper.service
systemctl status zookeeper.service

二、部署Kafka

2.1 准备安装包

wget -c https://mirrors.tuna.tsinghua.edu.cn/apache/kafka/2.6.0/kafka_2.13-2.6.0.tgz

2.2 安装解压

tar -xf kafka_2.13-2.6.0.tgz -C /opt/
cd /opt/
mv kafka_2.13-2.6.0 kafka

mkdir -p /mnt/cephfs/kafka0/logs /mnt/cephfs/kafka1/logs /mnt/cephfs/kafka2/logs

2.3 添加DNS域名解析或修改hosts文件

[root@lb03.host.com:/root]# vim /var/named/lowan.com.zone

kafka0          A       192.168.13.101
kafka1          A       192.168.13.102
kafka2          A       192.168.13.103

systemctl restart named

2.4 配置server.properties文件

vim /opt/kafka/config/server.properties
---------------------------------------->
kafka0的配置如下:
broker.id=0
listeners=PLAINTEXT://:9092
advertised.listeners=PLAINTEXT://kafka0.lowan.com:9092
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/mnt/cephfs/kafka0/logs
num.partitions=3
num.recovery.threads.per.data.dir=1
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
zookeeper.connect=zk01.lowan.com:2181,zk02.lowan.com:2181,zk03.lowan.com:2181
zookeeper.connection.timeout.ms=12000
group.initial.rebalance.delay.ms=0
group.max.session.timeout.ms=300000
group.min.session.timeout.ms=6000
auto.create.topics.enable=true
delete.topics.enable=true

---------------------------------------->
kafka1的配置如下:
broker.id=1
listeners=PLAINTEXT://:9092
advertised.listeners=PLAINTEXT://kafka1.lowan.com:9092
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/mnt/cephfs/kafka1/logs
num.partitions=3
num.recovery.threads.per.data.dir=1
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
zookeeper.connect=zk01.lowan.com:2181,zk02.lowan.com:2181,zk03.lowan.com:2181
zookeeper.connection.timeout.ms=12000
group.initial.rebalance.delay.ms=0
group.max.session.timeout.ms=300000
group.min.session.timeout.ms=6000
auto.create.topics.enable=true
delete.topics.enable=true

---------------------------------------->
kafka2的配置如下:
broker.id=2
listeners=PLAINTEXT://:9092
advertised.listeners=PLAINTEXT://kafka2.lowan.com:9092
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/mnt/cephfs/kafka2/logs
num.partitions=3
num.recovery.threads.per.data.dir=1
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
zookeeper.connect=zk01.lowan.com:2181,zk02.lowan.com:2181,zk03.lowan.com:2181
zookeeper.connection.timeout.ms=12000
group.initial.rebalance.delay.ms=0
group.max.session.timeout.ms=300000
group.min.session.timeout.ms=6000
auto.create.topics.enable=true
delete.topics.enable=true

2.5 启动kafka

#启动
/opt/kafka/bin/kafka-server-start.sh -daemon /opt/kafka/config/server.properties
#检查java进程
jps

ps -ef | grep kafka

#如停止
/opt/kafka/bin/kafka-server-stop.sh

2.6 添加守护进程

vim /etc/systemd/system/kafka.service
[Unit]
Description=kafka server
Requires=zookeeper.service
After=zookeeper.service

[Service]
Type=simple
User=root
Environment=JAVA_HOME=/usr/local/java
ExecStart=/bin/sh -c '/opt/kafka/bin/kafka-server-start.sh /opt/kafka/config/server.properties'
ExecStop=/bin/sh -c '/opt/kafka/bin/kafka-server-stop.sh'
Restart=on-failure
RestartSec=2s
SuccessExitStatus=0  143
[Install]
WantedBy=multi-user.target

:x 保存退出 

systemctl daemon-reload
systemctl enable kafka.service 
systemctl start kafka.service 
systemctl status kafka.service 

三、Kubernetes部署kafka-manager

3.1 准备docker镜像文件

方式一:直接pull
docker pull vimagick/cmak:3.0.0.5
docker tag vimagick/cmak:3.0.0.5 harbor.iot.com/kubernetes/kubernetes-kafkamanager:3.0.5
docker push harbor.iot.com/kubernetes/kubernetes-kafkamanager:3.0.5
方式二: 编辑Dockerfile
vim Dockerfile
内容如下:
FROM debian:buster

ENV SCALA_VERSION 2.12.10
ENV SBT_VERSION 1.3.8
ENV CMAK_VERSION 3.0.0.5

RUN set -xe \
    && apt update \
    && apt install -y openjdk-11-jre-headless curl wget \
    && wget -q https://downloads.lightbend.com/scala/$SCALA_VERSION/scala-$SCALA_VERSION.deb -O scala.deb \
    && wget -q https://dl.bintray.com/sbt/debian/sbt-$SBT_VERSION.deb -O sbt.deb \
    && dpkg -i scala.deb sbt.deb \
    && rm scala.deb sbt.deb \
    && rm -rf /var/lib/apt/lists/*

WORKDIR /opt/cmak

RUN set -xe \
    && mkdir src \
    && curl -sSL https://github.com/yahoo/CMAK/archive/$CMAK_VERSION.tar.gz | tar xz --strip 1 -C src \
    && cd src \
    && sbt clean universal:packageZipTarball \
    && cd .. \
    && tar xzf src/target/universal/cmak-$CMAK_VERSION.tgz --strip 1 \
    && rm -rf src

VOLUME /opt/cmak/conf

EXPOSE 9000

ENTRYPOINT ["bin/cmak"]
CMD ["-Dconfig.file=conf/application.conf", "-Dhttp.port=9000", "-Dpidfile.path=/dev/null"]

HEALTHCHECK CMD curl -f http://127.0.0.1/api/health || exit 1


:x 保存退出

docker build -t harbor.iot.com/kubernetes/kubernetes-kafkamanager:3.0.5 .

3.2 创建svc

[root@k8smaster01.host.com:/data/yaml/kafka-manager]# vim svc-kafkamanager.yaml

apiVersion: v1
kind: Service
metadata:
  name: kafka-manager
spec:
  ports:
  - protocol: TCP
    port: 9000
    targetPort: 9000
  selector:
    name: kafka-manager

3.3 创建ingressroute

[root@k8smaster01.host.com:/data/yaml/kafka-manager]# vim ingressroute-kafkamanager.yaml 
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  labels:
    name: kafka-manager
  name: kafka-manager-ingress
  annotations:
    traefik.ingress.kubernetes.io/router.entrypoints: web
    kubernetes.io/ingress.class: "traefik"
spec:
  entryPoints:
  - web
  routes:
  - match: Host(`cmak.lowan.com`) && PathPrefix(`/`)
    kind: Rule
    services:
    - name: kafka-manager
      port: 9000

3.4 创建deployment

[root@k8smaster01.host.com:/data/yaml/kafka-manager]# vim deployment-kafkamanager.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: kafka-manager
  labels:
    name: kafka-manager
spec:
  replicas: 1
  selector:
    matchLabels:
      name: kafka-manager
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1
      maxSurge: 1
  revisionHistoryLimit: 7
  progressDeadlineSeconds: 600
  template:
    metadata:
      labels:
        name: kafka-manager
    spec:
      containers:
      - name: kafka-manager
        image: harbor.iot.com/kubernetes/kubernetes-kafkamanager:3.0.0.5
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 9000
          protocol: TCP
        env:
        - name: ZK_HOSTS
          value: "zk01.lowan.com:2181,zk02.lowan.com:2181,zk03.lowan.com:2181"
        - name: KAFKA_MANAGER_USERNAME
          value: admin
        - name: KAFKA_MANAGER_PASSWORD
          value: admin
        resources:
          limits:
            cpu: 1500m
            memory: 4000Mi
          requests:
            cpu: 200m
            memory: 500Mi
      imagePullSecrets:
      - name: harbor
      terminationGracePeriodSeconds: 30
      securityContext:
        runAsUser: 0


3.5 添加DNS域名解析

[root@lb03.host.com:/root]# vim /var/named/lowan.com.zone
cmak            A       192.168.13.100

systemctl restart named

3.6 应用配置清单

kubectl apply -f svc-kafkamanager.yaml 
kubectl apply -f deployment-kafkamanager.yaml 
kubectl apply -f ingressroute-kafkamanager.yaml

3.7 访问网页并配置

访问:http://cmak.lowan.com/

添加kafka集群

在这里插入图片描述

`在这里插入图片描述

后续参考(集群):

01 kubernetes二进制部署
02 kubernetes辅助环境设置
03 K8S集群网络ACL规则
04 Ceph集群部署
05 部署zookeeper和kafka集群
06 部署日志系统
07 部署Indluxdb-telegraf
08 部署jenkins
09 部署k3s和Helm-Rancher
10 部署maven软件

转载至https://blog.csdn.net/weixin_43667733/article/details/117267272?ops_request_misc=%257B%2522request%255Fid%2522%253A%2522162917409816780271556748%2522%252C%2522scm%2522%253A%252220140713.130102334.pc%255Fall.%2522%257D&request_id=162917409816780271556748&biz_id=0&utm_medium=distribute.pc_search_result.none-task-blog-2~all~first_rank_ecpm_v1~rank_v29_ecpm-3-117267272.ecpm_v1_rank_v29&utm_term=k8s+Zookeeper%E5%92%8CKafka&spm=1018.2226.3001.4187

  • 0
    点赞
  • 0
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
好的,以下是一个基本的在k8s上部署zookeeper + kafka集群的配置教程: 1. 创建一个namespace 首先,我们需要创建一个namespace,用于部署zookeeperkafka集群。可以使用以下命令创建一个名为“zookeeper-kafka”的namespace: ``` kubectl create namespace zookeeper-kafka ``` 2. 部署Zookeeper 接下来,我们需要部署Zookeeper。可以使用以下YAML文件创建一个Zookeeper服务: ``` apiVersion: v1 kind: Service metadata: name: zookeeper spec: selector: app: zookeeper ports: - name: client port: 2181 protocol: TCP - name: follower port: 2888 protocol: TCP - name: leader port: 3888 protocol: TCP --- apiVersion: apps/v1 kind: StatefulSet metadata: name: zookeeper spec: serviceName: zookeeper replicas: 3 selector: matchLabels: app: zookeeper template: metadata: labels: app: zookeeper spec: containers: - name: zookeeper image: zookeeper:3.5.8 ports: - name: client containerPort: 2181 - name: follower containerPort: 2888 - name: leader containerPort: 3888 volumeMounts: - name: datadir mountPath: /data env: - name: ZOO_MY_ID valueFrom: fieldRef: fieldPath: metadata.name - name: ZOO_SERVERS value: zookeeper-0.zookeeper:2888:3888,zookeeper-1.zookeeper:2888:3888,zookeeper-2.zookeeper:2888:3888 volumeClaimTemplates: - metadata: name: datadir spec: accessModes: - ReadWriteOnce resources: requests: storage: 1Gi ``` 这将创建一个3个Pod的Zookeeper StatefulSet,并创建一个名为“zookeeper”的Service,暴露Zookeeper的客户端端口2181,follower端口2888和leader端口3888。 3. 部署Kafka 现在,我们可以部署Kafka。以下是一个Kafka部署的YAML文件示例: ``` apiVersion: v1 kind: Service metadata: name: kafka spec: type: NodePort selector: app: kafka ports: - name: kafka port: 9092 nodePort: 30092 protocol: TCP --- apiVersion: apps/v1 kind: StatefulSet metadata: name: kafka spec: serviceName: kafka replicas: 3 selector: matchLabels: app: kafka template: metadata: labels: app: kafka spec: containers: - name: kafka image: wurstmeister/kafka:2.13-2.7.0 ports: - name: kafka containerPort: 9092 env: - name: KAFKA_BROKER_ID valueFrom: fieldRef: fieldPath: metadata.name - name: KAFKA_ZOOKEEPER_CONNECT value: zookeeper-0.zookeeper:2181,zookeeper-1.zookeeper:2181,zookeeper-2.zookeeper:2181 - name: KAFKA_ADVERTISED_LISTENERS value: PLAINTEXT://$(hostname -f):9092 - name: KAFKA_LISTENERS value: PLAINTEXT://0.0.0.0:9092 - name: KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR value: "3" volumeMounts: - name: datadir mountPath: /data volumeClaimTemplates: - metadata: name: datadir spec: accessModes: - ReadWriteOnce resources: requests: storage: 1Gi ``` 这将创建一个3个Pod的Kafka StatefulSet和一个名为“kafka”的Service,它将Kafka的9092端口暴露为NodePort 30092。 4. 验证部署 现在,您可以使用以下命令检查ZookeeperKafka是否正在运行: ``` kubectl get pods -n zookeeper-kafka ``` 您应该看到3个Zookeeper和3个Kafka Pod处于“Running”状态。 接下来,您可以使用以下命令检查Kafka是否正在监听端口30092(或您自己选择的端口): ``` kubectl get services -n zookeeper-kafka ``` 您应该看到一个名为“kafka”的service,它将Kafka的9092端口暴露为30092端口。可以使用此端口测试Kafka是否正常运行。 至此,您已经成功地在k8s上部署zookeeper + kafka集群

“相关推荐”对你有帮助么?

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值