kafka集群-安装(本文以kafka 2.5.0版本为例)
kafka集群-准备
服务器结构
hostname | ip | 应用一 | 应用二 | kafka-broker.id |
node01 | 192.168.44.60 | kafka-2.5.0 | 0 | |
node02 | 192.168.44.61 | Zookeeper-3.5.6 | kafka-2.5.0 | 1 |
node03 | 192.168.44.62 | Zookeeper-3.5.6 | kafka-2.5.0 | 2 |
node04 | 192.168.44.63 | Zookeeper-3.5.6 |
kafka集群依赖于zookeeper,所以在搭建kafka的时候需要首先安装zookeeper
这里我就不再说明zookeeper的安装步骤了,大家可以参考Linux之Zookeeper集群安装详细说明
kafka集群-安装(本文以kafka 2.5.0版本为例)
1:下载并解压
下载地址:http://kafka.apache.org/downloads
cd /usr/local/
wget wget https://archive.apache.org/dist/kafka/2.5.0/kafka_2.13-2.5.0.tgz
tar -zxvf kafka_2.13-2.5.0.tgz
2:修改配置文件
[root@node01 config]# cd /usr/local/kafka_2.13-2.5.0/config
[root@node01 config]# ll
total 72
-rw-rw-r-- 1 root root 906 Apr 8 2020 connect-console-sink.properties
-rw-rw-r-- 1 root root 909 Apr 8 2020 connect-console-source.properties
-rw-rw-r-- 1 root root 5321 Apr 8 2020 connect-distributed.properties
-rw-rw-r-- 1 root root 883 Apr 8 2020 connect-file-sink.properties
-rw-rw-r-- 1 root root 881 Apr 8 2020 connect-file-source.properties
-rw-rw-r-- 1 root root 2247 Apr 8 2020 connect-log4j.properties
-rw-rw-r-- 1 root root 2540 Apr 8 2020 connect-mirror-maker.properties
-rw-rw-r-- 1 root root 2262 Apr 8 2020 connect-standalone.properties
-rw-rw-r-- 1 root root 1221 Apr 8 2020 consumer.properties
-rw-rw-r-- 1 root root 4675 Apr 8 2020 log4j.properties
-rw-rw-r-- 1 root root 1925 Apr 8 2020 producer.properties
-rw-rw-r-- 1 root root 6849 Apr 8 2020 server.properties
-rw-rw-r-- 1 root root 1032 Apr 8 2020 tools-log4j.properties
-rw-rw-r-- 1 root root 1169 Apr 8 2020 trogdor.conf
-rw-rw-r-- 1 root root 1205 Apr 8 2020 zookeeper.properties
[root@node01 config]# vi server.properties
这里修改/usr/local/kafka_2.13-2.5.0/config目录下载的server.properties配置文件
修改内容如下:
broker.id=0
listeners=PLAINTEXT://node01:9092
log.dirs=/home/logs
zookeeper.connect=node02:2181,node03:2181,node04:2181/kafka
说明:broker.id每个机器不一样,listeners监听,kafka组件之间通讯和每台机器对应即可,其他两个选项一致
将kafka-2.5.0-src所有文件分发到node02,和node03上并且分别修改broker.id,listeners
node02:broker.id=1
listeners=PLAINTEXT://node02:9092
node03:broker.id=2
listeners=PLAINTEXT://node03:9092
scp -r kafka-2.5.0-src node02:/usr/local/
scp -r kafka-2.5.0-src node03:/usr/local/
3:启动kafka
启动前要保证(1)zookeeper已经启动(2)确保防火墙关闭,各个node可以相互访问
cd /usr/local/kafka_2.13-2.5.0/bin
# 最后& 表示后台启动
./kafka-server-start.sh ../config/server.properties &
4:验证是否启动成功
[root@node01 kafka_2.13-2.5.0]# jps
3347 Jps
2943 Kafka
5:测试
5.1:创建topic: test-topic
cd /usr/local/kafka_2.13-2.5.0/bin
./kafka-topics.sh --zookeeper node02:2181,node03:2181,node04:2181/kafka --create --topic test-topic --partitions 2 --replication-factor 2
说明:--partitions 2 表示给test-topic创建两个分区,--replication-factor 2 表示给test-topic创建两个副本
5.2:查看已创建的topic列表
[root@node01 bin]# ./kafka-topics.sh --zookeeper node02:2181,node03:2181,node04:2181/kafka --list
test-topic
5.3:查看topic:test-topic的详情
./kafka-topics.sh --zookeeper node02:2181,node03:2181,node04:2181/kafka --describe --topic test-topic
5.4:在node01上启动生产者
cd /usr/local/kafka_2.13-2.5.0/bin
./kafka-console-producer.sh --broker-list node01:9092 --topic test-topic
5.5:在node02上启动消费者
cd /usr/local/kafka_2.13-2.5.0/bin
./kafka-console-consumer.sh --bootstrap-server node01:9092,node02:9092,node03:9092 --topic test-topic
5.6:node01发送消息,node02接收消息
至此kafka集群已经搭建成功
作者:慕菲烟云 欢迎转载,与人分享是进步的源泉!