Kafka集群安装

集群规划

主机	IP	zookeeper	kafka
centos7-1	192.168.3.51	zk1	kafka1
centos7-2	192.168.3.52	zk2	kafka2
centos7-3	192.168.3.53	zk3	kafka3

修改主机hosts文件

vi /etc/hosts

在文件最后添加内容：

192.168.3.51 kafka1 zk1
192.168.3.52 kafka2 zk2
192.168.3.53 kafka3 zk3

集群部署

下载地址：http://kafka.apache.org/downloads.html

部署

1. 上传安装包 kafka_2.12-3.0.0.tgz 到服务器

[vagrant@localhost ~]$ ll
total 276772
drwxrwxr-x. 8 vagrant vagrant       158 Jan  2 15:24 apache-zookeeper-3.5.7-bin
-rw-rw-r--. 1 vagrant vagrant   9311744 Jan  2 14:56 apache-zookeeper-3.5.7-bin.tar.gz
drwxrwxr-x. 8 vagrant vagrant        96 Jan  2 15:12 jdk-11
-rw-rw-r--. 1 vagrant vagrant  86486610 Jan 16 02:04 kafka_2.12-3.0.0.tgz
-rw-rw-r--. 1 vagrant vagrant 187611826 Jan  2 15:11 openjdk-11+28_linux-x64_bin.tar.gz

2. 解压安装包

[vagrant@localhost ~]$ tar -zxvf kafka_2.12-3.0.0.tgz

3. 解压后cd进入kafka目录

[vagrant@localhost ~]$ cd kafka_2.12-3.0.0/
[vagrant@localhost kafka_2.12-3.0.0]$ ll
total 64
drwxr-xr-x. 3 vagrant vagrant  4096 Sep  8 21:24 bin
drwxr-xr-x. 3 vagrant vagrant  4096 Sep  8 21:24 config
drwxr-xr-x. 2 vagrant vagrant  8192 Jan 16 02:04 libs
-rw-r--r--. 1 vagrant vagrant 14521 Sep  8 21:21 LICENSE
drwxr-xr-x. 2 vagrant vagrant   262 Sep  8 21:24 licenses
-rw-r--r--. 1 vagrant vagrant 28184 Sep  8 21:21 NOTICE
drwxr-xr-x. 2 vagrant vagrant    44 Sep  8 21:24 site-docs

4. 创建data目录

cd /home/vagrant/kafka_2.12-3.0.0
mkdir /home/vagrant/kafka_2.12-3.0.0/data

5. 修改配置文件：

vi server.properties

注意修改下面几项

以第二台机器为例

修改

broker.id=1
#listeners=PLAINTEXT://:9092
log.dirs=/tmp/kafka-logs
zookeeper.connect=localhost:2181

为：

broker.id=1
listeners=PLAINTEXT://192.168.3.51:9092
log.dirs=/home/vagrant/kafka_2.12-3.0.0/data
zookeeper.connect=192.168.3.51:2181,192.168.3.52:2181,192.168.3.53:2181/kafka

上面其中一项，listeners=PLAINTEXT://:9092 官方默认的配置文件中是注释掉的，也就是没有配置，集群是可以启动的，但是你去访问集群，比如创建一个topic：bin/kafka-topics.sh --bootstrap-server 192.168.3.51:9092 --create --partitions 1 --replication-factor 3 --topic first 就会超时报错：

Error while executing topic command : Timed out waiting for a node assignment. Call: createTopics
[2022-02-19 02:31:33,740] ERROR org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment. Call: createTopics
 (kafka.admin.TopicCommand$)

修改好配置后，cd进入bin目录启动kafka服务

cd /home/vagrant/kafka_2.12-3.0.0/bin

[vagrant@localhost bin]$ ll
total 160
-rwxr-xr-x. 1 vagrant vagrant  1423 Sep  8 21:21 connect-distributed.sh
-rwxr-xr-x. 1 vagrant vagrant  1396 Sep  8 21:21 connect-mirror-maker.sh
-rwxr-xr-x. 1 vagrant vagrant  1420 Sep  8 21:21 connect-standalone.sh
-rwxr-xr-x. 1 vagrant vagrant   861 Sep  8 21:21 kafka-acls.sh
-rwxr-xr-x. 1 vagrant vagrant   873 Sep  8 21:21 kafka-broker-api-versions.sh
-rwxr-xr-x. 1 vagrant vagrant   860 Sep  8 21:21 kafka-cluster.sh
-rwxr-xr-x. 1 vagrant vagrant   864 Sep  8 21:21 kafka-configs.sh
-rwxr-xr-x. 1 vagrant vagrant   945 Sep  8 21:21 kafka-console-consumer.sh
-rwxr-xr-x. 1 vagrant vagrant   944 Sep  8 21:21 kafka-console-producer.sh
-rwxr-xr-x. 1 vagrant vagrant   871 Sep  8 21:21 kafka-consumer-groups.sh
-rwxr-xr-x. 1 vagrant vagrant   948 Sep  8 21:21 kafka-consumer-perf-test.sh
-rwxr-xr-x. 1 vagrant vagrant   871 Sep  8 21:21 kafka-delegation-tokens.sh
-rwxr-xr-x. 1 vagrant vagrant   869 Sep  8 21:21 kafka-delete-records.sh
-rwxr-xr-x. 1 vagrant vagrant   866 Sep  8 21:21 kafka-dump-log.sh
-rwxr-xr-x. 1 vagrant vagrant   863 Sep  8 21:21 kafka-features.sh
-rwxr-xr-x. 1 vagrant vagrant   865 Sep  8 21:21 kafka-get-offsets.sh
-rwxr-xr-x. 1 vagrant vagrant   870 Sep  8 21:21 kafka-leader-election.sh
-rwxr-xr-x. 1 vagrant vagrant   863 Sep  8 21:21 kafka-log-dirs.sh
-rwxr-xr-x. 1 vagrant vagrant   873 Sep  8 21:21 kafka-metadata-shell.sh
-rwxr-xr-x. 1 vagrant vagrant   862 Sep  8 21:21 kafka-mirror-maker.sh
-rwxr-xr-x. 1 vagrant vagrant   959 Sep  8 21:21 kafka-producer-perf-test.sh
-rwxr-xr-x. 1 vagrant vagrant   874 Sep  8 21:21 kafka-reassign-partitions.sh
-rwxr-xr-x. 1 vagrant vagrant   874 Sep  8 21:21 kafka-replica-verification.sh
-rwxr-xr-x. 1 vagrant vagrant 10587 Sep  8 21:21 kafka-run-class.sh
-rwxr-xr-x. 1 vagrant vagrant  1376 Sep  8 21:21 kafka-server-start.sh
-rwxr-xr-x. 1 vagrant vagrant  1361 Sep  8 21:21 kafka-server-stop.sh
-rwxr-xr-x. 1 vagrant vagrant   860 Sep  8 21:21 kafka-storage.sh
-rwxr-xr-x. 1 vagrant vagrant   945 Sep  8 21:21 kafka-streams-application-reset.sh
-rwxr-xr-x. 1 vagrant vagrant   863 Sep  8 21:21 kafka-topics.sh
-rwxr-xr-x. 1 vagrant vagrant   879 Sep  8 21:21 kafka-transactions.sh
-rwxr-xr-x. 1 vagrant vagrant   958 Sep  8 21:21 kafka-verifiable-consumer.sh
-rwxr-xr-x. 1 vagrant vagrant   958 Sep  8 21:21 kafka-verifiable-producer.sh
-rwxr-xr-x. 1 vagrant vagrant  1714 Sep  8 21:21 trogdor.sh
drwxr-xr-x. 2 vagrant vagrant  4096 Sep  8 21:21 windows
-rwxr-xr-x. 1 vagrant vagrant   867 Sep  8 21:21 zookeeper-security-migration.sh
-rwxr-xr-x. 1 vagrant vagrant  1393 Sep  8 21:21 zookeeper-server-start.sh
-rwxr-xr-x. 1 vagrant vagrant  1366 Sep  8 21:21 zookeeper-server-stop.sh
-rwxr-xr-x. 1 vagrant vagrant  1019 Sep  8 21:21 zookeeper-shell.sh

6. 启动

/home/vagrant/kafka_2.12-3.0.0/bin/kafka-server-start.sh -daemon /home/vagrant/kafka_2.12-3.0.0/config/server.properties

[vagrant@localhost bin]$ jps
2434 QuorumPeerMain
6356 Jps
6333 Kafka

kafka服务启动成功

也可以查看日志

tail -f -n 500 /home/vagrant/kafka_2.12-3.0.0/logs/server.log

7. 停止

/home/vagrant/kafka_2.12-3.0.0/bin/kafka-server-stop.sh

配置环境变量

vi /etc/profile
----------------------------------------------------------------------------------
末尾增加如下内容(根据自己目录修改)：
export KAFKA_HOME=/home/vagrant/kafka_2.12-3.0.0
export PATH=$KAFKA_HOME/bin:$PATH
----------------------------------------------------------------------------------
 
source /etc/profile   # 使配置文件生效

环境变量配置好后，就可以在命令行任意位置执行 /home/vagrant/kafka_2.12-3.0.0/bin 下面的sh脚本了，不然需要带绝对路径。