kafka集群搭建

附：

        （1） Linux Centos7.5 三台虚拟机真实搭建Zookeeper集群

        （2）Linux Centos7.5 安装配置Oracle jdk1.8

ps:jdk版本不要过低。

我使用的是/kafka_2.13-2.4.1，linux centos7.5，jdk1.8.0_241

1、下载

附：http://kafka.apache.org/downloads

#下载
wget https://mirror.bit.edu.cn/apache/kafka/2.4.1/kafka_2.13-2.4.1.tgz

#解压
tar -zxvf kafka_2.13-2.4.1.tgz

#重命名
mv kafka_2.13-2.4.1 kafka

ps:三台虚拟机均安装。

2、编辑server.properties

[root@localhost kafka]# cd config/
[root@localhost config]# ll
total 72
-rw-r--r--. 1 root root  906 Mar  2 16:32 connect-console-sink.properties
-rw-r--r--. 1 root root  909 Mar  2 16:32 connect-console-source.properties
-rw-r--r--. 1 root root 5321 Mar  2 16:32 connect-distributed.properties
-rw-r--r--. 1 root root  883 Mar  2 16:32 connect-file-sink.properties
-rw-r--r--. 1 root root  881 Mar  2 16:32 connect-file-source.properties
-rw-r--r--. 1 root root 2247 Mar  2 16:32 connect-log4j.properties
-rw-r--r--. 1 root root 1539 Mar  2 16:32 connect-mirror-maker.properties
-rw-r--r--. 1 root root 2262 Mar  2 16:32 connect-standalone.properties
-rw-r--r--. 1 root root 1221 Mar  2 16:32 consumer.properties
-rw-r--r--. 1 root root 4675 Mar  2 16:32 log4j.properties
-rw-r--r--. 1 root root 1925 Mar  2 16:32 producer.properties
-rw-r--r--. 1 root root 6848 Mar  2 16:32 server.properties
-rw-r--r--. 1 root root 1032 Mar  2 16:32 tools-log4j.properties
-rw-r--r--. 1 root root 1169 Mar  2 16:32 trogdor.conf
-rw-r--r--. 1 root root 1205 Mar  2 16:32 zookeeper.properties
[root@localhost config]# vim server.properties

server.properties：

broker.id=1

log.dirs=/usr/local/kafka/logs

listeners=PLAINTEXT://192.168.130:9092

zookeeper.connect=192.168.160.130:2181,192.168.160.133:2181,192.168.160.134:2181

ps:三台虚拟机均修改以上内容。192.168.160.130:2181   broker.id=0； 192.168.160.133:2181 broker.id=1；192.168.160.134:2181  broker.id=2。listeners均改成当前服务器的ip。

问题一：Uninitialized object exists on backward branch 79

[root@iZ8vbh7c4195en5tcpx559Z kafka_2.12]# bin/kafka-server-start.sh config/server.properties
[2020-03-24 20:30:21,667] INFO Registered kafka:type=kafka.Log4jController MBean (kafka.utils.Log4jControllerRegistration$)
[2020-03-24 20:30:22,161] ERROR Exiting Kafka due to fatal exception (kafka.Kafka$)
java.lang.VerifyError: Uninitialized object exists on backward branch 79
Exception Details:
  Location:
    scala/collection/immutable/HashMap$HashTrieMap.split()Lscala/collection/immutable/Seq; @249: goto
  Reason:
    Error exists in the bytecode
  Bytecode:
    0000000: 2ab6 0064 04a0 001e b200 c1b2 00c6 04bd
    0000010: 0002 5903 2a53 c000 c8b6 00cc b600 d0c0
    0000020: 00d2 b02a b600 38b8 0042 3c1b 04a4 0156
    0000030: 1b05 6c3d 2a1b 056c 2ab6 0038 b700 d43e
    0000040: 2ab6 0038 021d 787e 3604 2ab6 0038 0210
    0000050: 201d 647c 7e36 05bb 0019 59b2 00c6 2ab6
    0000060: 003a c000 c8b6 00d8 b700 db1c b600 df3a
    0000070: 0619 06c6 001a 1906 b600 e3c0 008b 3a07

解决：升级jdk版本,原版本：1.8.0_11，此处我是升级到了1.8.0_241

[root@localhost config]# java -version
java version "1.8.0_11"
Java(TM) SE Runtime Environment (build 1.8.0_11-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.11-b03, mixed mode)
[root@localhost config]# cd ..
[root@localhost kafka]# cd ..
[root@localhost local]# ls
bin                        jdk1.8.0_11                 nacos        share
elasticsearch-7.4.2        jdk-8u241-linux-x64.tar.gz  redis        src
elasticsearch-head-master  kafka                       redis-5.0.3  zookeeper
etc                        lib                         rocketmq
games                      lib64                       sbin
include                    libexec                     seata
[root@localhost local]# tar -zxvf jdk-8u241-linux-x64.tar.gz
[root@localhost local]# vim /etc/profile
######修改java home的路径

[root@localhost ~]# java -version
-bash: /usr/local/jdk1.8.0_241/bin/java: Permission denied

问题：-bash: /usr/local/jdk1.8.0_241/bin/java: Permission denied

解决：赋予权限

[root@localhost ~]# java -version
-bash: /usr/local/jdk1.8.0_241/bin/java: Permission denied
[root@localhost ~]# chmod 777 /usr/local/jdk1.8.0_241/bin/java
[root@localhost ~]# java -version
java version "1.8.0_241"
Java(TM) SE Runtime Environment (build 1.8.0_241-b07)
Java HotSpot(TM) 64-Bit Server VM (build 25.241-b07, mixed mode)

3、启动

[root@localhost kafka]# cd bin
#前台启动
[root@localhost bin]# ./kafka-server-start.sh ../config/server.properties
#or 后台启动
[root@localhost bin]# ./kafka-server-start.sh -daemon  ../config/server.properties
[root@localhost bin]# jps
5810 Jps
1640 QuorumPeerMain
5768 Kafka

4、验证是否创建成功

   4.1 创建topic

[root@localhost bin]# ./kafka-topics.sh --create --zookeeper 192.168.160.130:2181,192.168.160.133:2181,192.168.160.134:2181 --partitions 3 --replication-factor 3 --topic test
Created topic test.

  4.2 打开任意虚拟机的shell界面,模拟生产者发送消息：

[root@localhost bin]# ./kafka-console-producer.sh --broker-list 192.168.160.130:9092,192.168.160.133:9092,192.168.160.134:9092 --topic test
>hello kafka
>

4.3 打开任意虚拟机的shell界面,模拟消费者发送消息：

[root@localhost bin]# ./kafka-console-consumer.sh --bootstrap-server 192.168.160.130:9092,192.168.160.133:9092,192.168.160.134:9092 --topic test --from-beginning
hello kafka