linux下搭建zookeeper+kafka集群

linux版本:centos7.2

链接:https://pan.baidu.com/s/1jjHoz_GQNe9pwTT40ZWryg
提取码:4mtp

kafka版本:kafka_2.12-2.7.0.tgz

链接:https://pan.baidu.com/s/1b5yOLpyPh5rQy9hEGDq2Ag
提取码:ciai

zookeeper环境:

https://blog.csdn.net/weixin_43914685/article/details/113762882

1.解压安装包

[root@slave1 local]# tar -zxvf kafka_2.12-2.7.0.tgz 

2.创建软连接

[root@slave1 local]# ln -s kafka_2.12-2.7.0 kafka0
[root@slave1 local]# ls
apache-tomcat-8.5.63         bin  games    kafka0            kafka_2.12-2.7.0.tgz  lib64    lz-server  share     src
apache-tomcat-8.5.63.tar.gz  etc  include  kafka_2.12-2.7.0  lib                   libexec  sbin       software

3.修改kafka配置文件

[root@master local]# vim kafka0/config/server.properties 

3台机器分别修改id为0 1 2,listeners修改为自己的ip地址

broker.id=0

############################# Socket Server Settings #############################

# The address the socket server listens on. It will get the value returned from 
# java.net.InetAddress.getCanonicalHostName() if not configured.
#   FORMAT:
#     listeners = listener_name://host_name:port
#   EXAMPLE:
#     listeners = PLAINTEXT://your.host.name:9092
listeners=PLAINTEXT://192.168.1.108:9092

4.日志生成目录

log.dirs=/tmp/kafka-logs

5.设置Zookeeper集群地址

zookeeper.connect=192.168.1.108:2181,192.168.1.109:2181,192.168.1.110:2181

6.Topic的默认Partition数量

num.partitions 为新建Topic的默认Partition数量,partition数量提升,一定程度上可以提升并发性

num.partitions=3

offsets和transaction
两个topic,分组元数据的复制因子,为了保证可用性,在生产上建议设置大于1。

default.replication.factor为kafka保存消息的副本数,如果一个副本失效了,另一个还可以继续提供服务,是在自动创建topic时的默认副本数,可以设置为3

############################# Internal Topic Settings  #############################
# The replication factor for the group metadata internal topics "__consumer_offsets" and "__transaction_state"
# For anything other than development testing, a value greater than 1 is recommended to ensure availability such as 3.
offsets.topic.replication.factor=3
transaction.state.log.replication.factor=3
transaction.state.log.min.isr=3

7.启动kafka

[root@slave2 kafka0]# bin/kafka-server-start.sh -daemon config/server.properties

8.查看日志

进入kafka的log目录,结尾输出日志started 启动成功

[root@slave2 logs]# cd logs/
[root@slave2 logs]# cat kafkaServer.out 
[2021-02-13 06:48:06,563] INFO [KafkaServer id=2] started (kafka.server.KafkaServer)
[2021-02-13 06:48:09,168] INFO [broker-2-to-controller-send-thread]: Recorded new controller, from now on will use broker 2 (kafka.server.BrokerToControllerRequestThread)

9.查看进程

[root@slave2 logs]# jps
2768 Jps
2248 QuorumPeerMain
2682 Kafka

10.停止kafka命令

[root@slave2 logs]# bin/kafka-server-stop.sh config/server.properties

11.配置环境变量

[root@master ~]# vim /etc/profile
[root@master ~]# source /etc/profile
[root@master ~]# echo $KAFKA_HOME

12.kafka和Zookeeper已启动完成

  1. 创建topic集群,3台机器可以同步接收消息,一台挂掉不影响另外两台继续工作
[root@master ~]# kafka-topics.sh --create --zookeeper '192.168.1.108:2181,192.168.1.109:2181,192.168.1.110:2181' --replication-factor 3 --partitions 3 --topic kafka0
Created topic kafka0.
  1. 查看主题
[root@master ~]# kafka-topics.sh --list --zookeeper 192.168.1.108:2181
__consumer_offsets
kafka0
  1. 发送消息

通过192.168.1.109给192.168.1.108发送消息

[root@master kafka1]# bin/kafka-console-producer.sh --broker-list 192.168.1.108:9092 --topic test
  1. 接收消息

192.168.1.108主机接收消息

[root@master ~]# kafka-console-consumer.sh --bootstrap-server 192.168.1.108:9092 --topic kafka0 --from-beginning
hello
  1. 查看主题
[root@master ~]# kafka-topics.sh --zookeeper localhost:2181 --describe  --topic kafka0
Topic: kafka0   PartitionCount: 3       ReplicationFactor: 3    Configs: 
        Topic: kafka0   Partition: 0    Leader: 1       Replicas: 1,2,0 Isr: 1,2,0
        Topic: kafka0   Partition: 1    Leader: 2       Replicas: 2,0,1 Isr: 2,0,1
        Topic: kafka0   Partition: 2    Leader: 0       Replicas: 0,1,2 Isr: 0,1,2

从中可以看到,test主题分了三个区,复制因子是3。

  1. 删除主题
[root@master ~]# kafka-topics.sh --zookeeper localhost:2181 --delete  --topic test
Topic test is marked for deletion.
Note: This will have no impact if delete.topic.enable is not set to true.
  • 2
    点赞
  • 2
    收藏
    觉得还不错? 一键收藏
  • 1
    评论
好的,以下是一个基本的在k8s上部署zookeeper + kafka集群的配置教程: 1. 创建一个namespace 首先,我们需要创建一个namespace,用于部署zookeeperkafka集群。可以使用以下命令创建一个名为“zookeeper-kafka”的namespace: ``` kubectl create namespace zookeeper-kafka ``` 2. 部署Zookeeper 接下来,我们需要部署Zookeeper。可以使用以下YAML文件创建一个Zookeeper服务: ``` apiVersion: v1 kind: Service metadata: name: zookeeper spec: selector: app: zookeeper ports: - name: client port: 2181 protocol: TCP - name: follower port: 2888 protocol: TCP - name: leader port: 3888 protocol: TCP --- apiVersion: apps/v1 kind: StatefulSet metadata: name: zookeeper spec: serviceName: zookeeper replicas: 3 selector: matchLabels: app: zookeeper template: metadata: labels: app: zookeeper spec: containers: - name: zookeeper image: zookeeper:3.5.8 ports: - name: client containerPort: 2181 - name: follower containerPort: 2888 - name: leader containerPort: 3888 volumeMounts: - name: datadir mountPath: /data env: - name: ZOO_MY_ID valueFrom: fieldRef: fieldPath: metadata.name - name: ZOO_SERVERS value: zookeeper-0.zookeeper:2888:3888,zookeeper-1.zookeeper:2888:3888,zookeeper-2.zookeeper:2888:3888 volumeClaimTemplates: - metadata: name: datadir spec: accessModes: - ReadWriteOnce resources: requests: storage: 1Gi ``` 这将创建一个3个Pod的Zookeeper StatefulSet,并创建一个名为“zookeeper”的Service,暴露Zookeeper的客户端端口2181,follower端口2888和leader端口3888。 3. 部署Kafka 现在,我们可以部署Kafka。以下是一个Kafka部署的YAML文件示例: ``` apiVersion: v1 kind: Service metadata: name: kafka spec: type: NodePort selector: app: kafka ports: - name: kafka port: 9092 nodePort: 30092 protocol: TCP --- apiVersion: apps/v1 kind: StatefulSet metadata: name: kafka spec: serviceName: kafka replicas: 3 selector: matchLabels: app: kafka template: metadata: labels: app: kafka spec: containers: - name: kafka image: wurstmeister/kafka:2.13-2.7.0 ports: - name: kafka containerPort: 9092 env: - name: KAFKA_BROKER_ID valueFrom: fieldRef: fieldPath: metadata.name - name: KAFKA_ZOOKEEPER_CONNECT value: zookeeper-0.zookeeper:2181,zookeeper-1.zookeeper:2181,zookeeper-2.zookeeper:2181 - name: KAFKA_ADVERTISED_LISTENERS value: PLAINTEXT://$(hostname -f):9092 - name: KAFKA_LISTENERS value: PLAINTEXT://0.0.0.0:9092 - name: KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR value: "3" volumeMounts: - name: datadir mountPath: /data volumeClaimTemplates: - metadata: name: datadir spec: accessModes: - ReadWriteOnce resources: requests: storage: 1Gi ``` 这将创建一个3个Pod的Kafka StatefulSet和一个名为“kafka”的Service,它将Kafka9092端口暴露为NodePort 30092。 4. 验证部署 现在,您可以使用以下命令检查ZookeeperKafka是否正在运行: ``` kubectl get pods -n zookeeper-kafka ``` 您应该看到3个Zookeeper和3个Kafka Pod处于“Running”状态。 接下来,您可以使用以下命令检查Kafka是否正在监听端口30092(或您自己选择的端口): ``` kubectl get services -n zookeeper-kafka ``` 您应该看到一个名为“kafka”的service,它将Kafka9092端口暴露为30092端口。可以使用此端口测试Kafka是否正常运行。 至此,您已经成功地在k8s上部署了zookeeper + kafka集群

“相关推荐”对你有帮助么?

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论 1
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值