Kafka集群配置
参考:参考文章
https://zhuanlan.zhihu.com/p/136758040?from_voters_page=true
因为kafka集群要配合zookeeper集群一起使用,所以要先搭建zookeeper集群
1、搭建zookeeper集群
首先下载zookeeper,官网下载,建议镜像网站下载(速度快)
下载完成之后解压
tar -zxvf 对应的zookeeper的压缩包
将zookeeper复制一份,命名为zookeeper-1
将zookeeper-1中的zoo_sample.cfg文件复制一份改名为: zoo.cfg
修改zoo.cfg文件
-
修改端口:clientPort=2181
-
修改数据目录:dataDir=/opt/zookeeper-1/datalog
-
增加以下配置:
server.1=localhost.:2887:3887 server.2=localhost.:2888:3888 server.3=localhost.:2889:3889 admin.serverPort=8000
在根据zookeeper-1复制两份,分别命名为zookeeper-2、zookeeper-3
修改zookeeper-2、zookeeper-3中的zoo.cfg文件
- 修改zookeeper-2的zoo.cfg中clientPort=2182,dataDir=/opt/zookeeper-2/datalog
- 修改zookeeper3
的
zoo.cfg中clientPort=2183,dataDir=/opt/zookeeper-3/datalog
创建刚才在配置文件中写的目录
mkdir /opt/zookeeper-1/datalog
mkdir /opt/zookeeper-2/datalog
mkdir /opt/zookeeper-3/datalog
分别在对应的datalog目录下执行以下命令,写入myid。
echo "1" > /opt/zookeeper-1/datalog/myid
echo "2" > /opt/zookeeper-2/datalog/myid
echo "3" > /opt/zookeeper-3/datalog/myid
最后分别启动zookeeper集群
/opt/zookeeper-1/bin/zkServer.sh start
/opt/zookeeper-2/bin/zkServer.sh start
/opt/zookeeper-3/bin/zkServer.sh start
使用如下命令判断是否启动成功
注意如果只启动一个zookeeper,就运行对应的检查状态命令,会提示zookeeper可能启动成功,只要运行了多个就不会提示这个信息
/opt/zookeeper-1/bin/zkServer.sh status
/opt/zookeeper-2/bin/zkServer.sh status
/opt/zookeeper-3/bin/zkServer.sh status
搭建Kafka集群
下载
kafka官网
开始安装
解压
tar -zxvf kafka_xxx.x.x.tgz
将config/server.properties复制三份,分别命名为server1.properties,
server2.properties,
server3.properties
本地VMware配置:
-
修改
server1.properties
-
- broker.id=1
- listeners=PLAINTEXT://:9092
- advertised.listeners=PLAINTEXT://本机IP:9092
- log.dirs=/opt/kafka/logs/kafka1-logs
- zookeeper.connect=localhost:2181,localhost:2182,localhost:2183
-
同理,修改
server2.properties
-
- broker.id=2
- listeners=PLAINTEXT://:9093
- advertised.listeners=PLAINTEXT://本机IP:9093
- log.dirs=/opt/kafka/logs/kafka2-logs
- zookeeper.connect=localhost:2181,localhost:2182,localhost:2183
-
同理,修改
server3.properties
-
- broker.id=3
- listeners=PLAINTEXT://:9094
- advertised.listeners=PLAINTEXT://本机IP:9094
- log.dirs=/ashura/kafka_2.11-2.2.1/logs/kafka3-logs
- zookeeper.connect=localhost:2181,localhost:2182,localhost:2183
阿里云服务器配置:
-
修改
server1.properties
- port=9092
- host.name=阿里云内网地址
- advertised.host.name=阿里云外网映射地址:9092
-
- broker.id=1
- listeners=PLAINTEXT://:9092
- log.dirs=/opt/kafka/logs/kafka1-logs
- zookeeper.connect=localhost:2181,localhost:2182,localhost:2183
-
同理,修改
server2.properties
-
port=9093
-
host.name=阿里云内网地址
-
advertised.host.name=阿里云外网映射地址:9093
-
-
- broker.id=2
- listeners=PLAINTEXT://:9093
- log.dirs=/opt/kafka/logs/kafka2-logs
- zookeeper.connect=localhost:2181,localhost:2182,localhost:2183
-
同理,修改
server3.properties
- port=9094
- host.name=阿里云内网地址
- advertised.host.name=阿里云外网映射地址:9094
-
- broker.id=3
- listeners=PLAINTEXT://:9094
- log.dirs=/ashura/kafka_2.11-2.2.1/logs/kafka3-logs
- zookeeper.connect=localhost:2181,localhost:2182,localhost:2183
先启动zookeeper集群:
/opt/zookeeper-1/bin/zkServer.sh start
/opt/zookeeper-2/bin/zkServer.sh start
/opt/zookeeper-3/bin/zkServer.sh start
启动kafka集群:(建议第一次不要后台启动)
/opt/kafka/bin/kafka-server-start.sh /opt/kafka/config/server1.properties
/opt/kafka/bin/kafka-server-start.sh /opt/kafka/config/server2.properties
/opt/kafka/bin/kafka-server-start.sh /opt/kafka/config/server3.properties
创建主题:
/opt/kafka/bin/kafka-topics.sh --bootstrap-server 127.0.0.1:9092 --create --topic topicname --replication-factor 3 --partitions 3
启动生产者:
/opt/kafka/bin/kafka-console-producer.sh --bootstrap-server 127.0.0.1:9092 --topic topicname
启动消费者:
/opt/kafka/bin/kafka-console-consumer.sh --bootstrap-server 127.0.0.1:9092 --topic topicname
测试:
ucer.sh --bootstrap-server 127.0.0.1:9092 --topic topicname
启动消费者:
/opt/kafka/bin/kafka-console-consumer.sh --bootstrap-server 127.0.0.1:9092 --topic topicname
测试:
在生产者端输入信息消费者端能看到对应的输入信息