kafka学习记录—安装配置和脚本测试

柒柚jun

已于 2022-04-22 22:16:18 修改

阅读量2k

点赞数 1

文章标签： kafka linux

于 2022-04-22 22:12:57 首次发布

本文链接：https://blog.csdn.net/qq_53320067/article/details/124356079

版权

整体进展
1. 下载插件
2. 创建目录
3. 解压kafka，修改三台集群配置文件
4. 添加kafka环境变量
5. 下载hadoop，添加环境变量
6. ssh登录
7. 启动zookeeper，修改zookeeper配置信息
8. 编写zk.sh脚本
9. 编写测试xsync脚本，分发kafka成功
10. topic命令操作，增加主题，查看详情信息
11. 生产者、消费者实战操作（发送数据、接收数据）
主要问题
1. 创建主题时报错超时，未能创建成功，查阅资料后发现将端口号弄混，在配置文件中未设置listeners，则命令操作主机名对应的端口号默认为9092，而zookeeper中设置的是配置kafka要连接集群的地址，修改后创建成功。

kafka—Topic命令操作

下载插件

 yum install -y epel-release

关闭防火墙

[root@hadoop100 ~] systemctl stop firewalld
[root@hadoop100 ~] systemctl disable firewalld.service
Removed symlink /etc/systemd/system/multi-user.target.wants/firewalld.service.
Removed symlink /etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service.

创建module和software文件夹，并把他们设置在用户qiyou组下

[root@hadoop100 ~] mkdir /opt/module
[root@hadoop100 ~] mkdir /opt/software
[root@hadoop100 ~] cd /opt/
[root@hadoop100 opt] ll
总用量 0
drwx--x--x. 4 root root 28 4月  14 19:16 containerd
drwxr-xr-x. 2 root root  6 4月  21 13:56 module
drwxr-xr-x. 2 root root  6 10月 31 2018 rh
drwxr-xr-x. 3 root root 30 4月  20 16:23 soft
drwxr-xr-x. 2 root root  6 4月  21 13:56 software

[root@hadoop100 opt] chown qiyou:qiyou /opt/module
[root@hadoop100 opt] vim /etc/sudoers
[root@hadoop100 opt] vim /etc/sudoers
[root@hadoop100 opt] chown qiyou:qiyou /opt/module
[root@hadoop100 opt] chown qiyou:qiyou /opt/software/
[root@hadoop100 opt] ll
总用量 0
drwx--x--x. 4 root  root  28 4月  14 19:16 containerd
drwxr-xr-x. 2 qiyou qiyou  6 4月  21 13:56 module
drwxr-xr-x. 2 root  root   6 10月 31 2018 rh
drwxr-xr-x. 3 root  root  30 4月  20 16:23 soft
drwxr-xr-x. 2 qiyou qiyou  6 4月  21 13:56 software

#切换用户
[root@hadoop101 ~] su qiyou
[qiyou@hadoop101 root]$ 
[qiyou@hadoop101 root]$ cd /opt/
[qiyou@hadoop101 opt]$ ll
总用量 0
drwx--x--x. 4 root  root  28 4月  14 19:16 containerd
drwxr-xr-x. 2 qiyou qiyou  6 4月  21 13:56 module
drwxr-xr-x. 2 root  root   6 10月 31 2018 rh
drwxr-xr-x. 3 root  root  30 4月  20 16:23 soft
drwxr-xr-x. 2 qiyou qiyou  6 4月  21 13:56 software
[qiyou@hadoop101 opt]$ cd software/
[qiyou@hadoop101 software]$ ll
总用量 84460
-rw-r--r--. 1 root root 86486610 4月  21 14:27 kafka_2.12-3.0.0.tgz

#解压kafka
[qiyou@hadoop101 software]$ tar -vxzf kafka_2.12-3.0.0.tgz -C /opt/module/

kafka_2.12-3.0.0/libs/commons-lang3-3.8.1.jar
kafka_2.12-3.0.0/libs/connect-file-3.0.0.jar
kafka_2.12-3.0.0/libs/connect-basic-auth-extension-3.0.0.jar
kafka_2.12-3.0.0/libs/connect-mirror-3.0.0.jar
kafka_2.12-3.0.0/libs/connect-mirror-client-3.0.0.jar
kafka_2.12-3.0.0/libs/kafka-streams-3.0.0.jar
kafka_2.12-3.0.0/libs/rocksdbjni-6.19.3.jar
kafka_2.12-3.0.0/libs/kafka-streams-scala_2.12-3.0.0.jar
kafka_2.12-3.0.0/libs/kafka-streams-test-utils-3.0.0.jar
kafka_2.12-3.0.0/libs/kafka-streams-examples-3.0.0.jar

修改kafka名字

[qiyou@hadoop101 software]$ cd /opt/module/
[qiyou@hadoop101 module]$ ll
总用量 0
drwxr-xr-x. 7 qiyou qiyou 105 9月   9 2021 kafka_2.12-3.0.0
[qiyou@hadoop101 module]$ mv kafka_2.12-3.0.0/ kafka
[qiyou@hadoop101 module]$ ll
总用量 0
drwxr-xr-x. 7 qiyou qiyou 105 9月   9 2021 kafka
[qiyou@hadoop101 module]$ cd kafka/
[qiyou@hadoop101 kafka]$ ll
总用量 64
drwxr-xr-x. 3 qiyou qiyou  4096 9月   9 2021 bin
drwxr-xr-x. 3 qiyou qiyou  4096 9月   9 2021 config
drwxr-xr-x. 2 qiyou qiyou  8192 4月  21 14:31 libs
-rw-r--r--. 1 qiyou qiyou 14521 9月   9 2021 LICENSE
drwxr-xr-x. 2 qiyou qiyou   262 9月   9 2021 licenses
-rw-r--r--. 1 qiyou qiyou 28184 9月   9 2021 NOTICE
drwxr-xr-x. 2 qiyou qiyou    44 9月   9 2021 site-docs

查看kafka脚本文件

[qiyou@hadoop101 kafka]$ cd bin/
[qiyou@hadoop101 bin]$ ll
总用量 160
-rwxr-xr-x. 1 qiyou qiyou  1423 9月   9 2021 connect-distributed.sh
-rwxr-xr-x. 1 qiyou qiyou  1396 9月   9 2021 connect-mirror-maker.sh
-rwxr-xr-x. 1 qiyou qiyou  1420 9月   9 2021 connect-standalone.sh
-rwxr-xr-x. 1 qiyou qiyou   861 9月   9 2021 kafka-acls.sh
-rwxr-xr-x. 1 qiyou qiyou   873 9月   9 2021 kafka-broker-api-versions.sh
-rwxr-xr-x. 1 qiyou qiyou   860 9月   9 2021 kafka-cluster.sh
-rwxr-xr-x. 1 qiyou qiyou   864 9月   9 2021 kafka-configs.sh
-rwxr-xr-x. 1 qiyou qiyou   945 9月   9 2021 kafka-console-consumer.sh
-rwxr-xr-x. 1 qiyou qiyou   944 9月   9 2021 kafka-console-producer.sh
-rwxr-xr-x. 1 qiyou qiyou   871 9月   9 2021 kafka-consumer-groups.sh
-rwxr-xr-x. 1 qiyou qiyou   948 9月   9 2021 kafka-consumer-perf-test.sh
-rwxr-xr-x. 1 qiyou qiyou   871 9月   9 2021 kafka-delegation-tokens.sh
-rwxr-xr-x. 1 qiyou qiyou   869 9月   9 2021 kafka-delete-records.sh
-rwxr-xr-x. 1 qiyou qiyou   866 9月   9 2021 kafka-dump-log.sh
-rwxr-xr-x. 1 qiyou qiyou   863 9月   9 2021 kafka-features.sh
-rwxr-xr-x. 1 qiyou qiyou   865 9月   9 2021 kafka-get-offsets.sh
-rwxr-xr-x. 1 qiyou qiyou   870 9月   9 2021 kafka-leader-election.sh
-rwxr-xr-x. 1 qiyou qiyou   863 9月   9 2021 kafka-log-dirs.sh
-rwxr-xr-x. 1 qiyou qiyou   873 9月   9 2021 kafka-metadata-shell.sh
-rwxr-xr-x. 1 qiyou qiyou   862 9月   9 2021 kafka-mirror-maker.sh
-rwxr-xr-x. 1 qiyou qiyou   959 9月   9 2021 kafka-producer-perf-test.sh
-rwxr-xr-x. 1 qiyou qiyou   874 9月   9 2021 kafka-reassign-partitions.sh
-rwxr-xr-x. 1 qiyou qiyou   874 9月   9 2021 kafka-replica-verification.sh
-rwxr-xr-x. 1 qiyou qiyou 10587 9月   9 2021 kafka-run-class.sh
-rwxr-xr-x. 1 qiyou qiyou  1376 9月   9 2021 kafka-server-start.sh
-rwxr-xr-x. 1 qiyou qiyou  1361 9月   9 2021 kafka-server-stop.sh
-rwxr-xr-x. 1 qiyou qiyou   860 9月   9 2021 kafka-storage.sh
-rwxr-xr-x. 1 qiyou qiyou   945 9月   9 2021 kafka-streams-application-reset.sh
-rwxr-xr-x. 1 qiyou qiyou   863 9月   9 2021 kafka-topics.sh
-rwxr-xr-x. 1 qiyou qiyou   879 9月   9 2021 kafka-transactions.sh
-rwxr-xr-x. 1 qiyou qiyou   958 9月   9 2021 kafka-verifiable-consumer.sh
-rwxr-xr-x. 1 qiyou qiyou   958 9月   9 2021 kafka-verifiable-producer.sh
-rwxr-xr-x. 1 qiyou qiyou  1714 9月   9 2021 trogdor.sh
drwxr-xr-x. 2 qiyou qiyou  4096 9月   9 2021 windows
-rwxr-xr-x. 1 qiyou qiyou   867 9月   9 2021 zookeeper-security-migration.sh
-rwxr-xr-x. 1 qiyou qiyou  1393 9月   9 2021 zookeeper-server-start.sh
-rwxr-xr-x. 1 qiyou qiyou  1366 9月   9 2021 zookeeper-server-stop.sh
-rwxr-xr-x. 1 qiyou qiyou  1019 9月   9 2021 zookeeper-shell.sh
[qiyou@hadoop101 bin]$ cd ..

查看kafka配置文件

[qiyou@hadoop101 kafka]$ cd config/
[qiyou@hadoop101 config]$ ll
总用量 72
-rw-r--r--. 1 qiyou qiyou  906 9月   9 2021 connect-console-sink.properties
-rw-r--r--. 1 qiyou qiyou  909 9月   9 2021 connect-console-source.properties
-rw-r--r--. 1 qiyou qiyou 5475 9月   9 2021 connect-distributed.properties
-rw-r--r--. 1 qiyou qiyou  883 9月   9 2021 connect-file-sink.properties
-rw-r--r--. 1 qiyou qiyou  881 9月   9 2021 connect-file-source.properties
-rw-r--r--. 1 qiyou qiyou 2103 9月   9 2021 connect-log4j.properties
-rw-r--r--. 1 qiyou qiyou 2540 9月   9 2021 connect-mirror-maker.properties
-rw-r--r--. 1 qiyou qiyou 2262 9月   9 2021 connect-standalone.properties
-rw-r--r--. 1 qiyou qiyou 1221 9月   9 2021 consumer.properties
drwxr-xr-x. 2 qiyou qiyou  102 9月   9 2021 kraft
-rw-r--r--. 1 qiyou qiyou 4674 9月   9 2021 log4j.properties
-rw-r--r--. 1 qiyou qiyou 1925 9月   9 2021 producer.properties
-rw-r--r--. 1 qiyou qiyou 6849 9月   9 2021 server.properties
-rw-r--r--. 1 qiyou qiyou 1032 9月   9 2021 tools-log4j.properties
-rw-r--r--. 1 qiyou qiyou 1169 9月   9 2021 trogdor.conf
-rw-r--r--. 1 qiyou qiyou 1205 9月   9 2021 zookeeper.properties

修改配置文件，即以下三个参数，三台集群的broker.id必须不一样，是唯一标识的id，并修改zookeeper中配置kafka要连接集群的地址
listener未配置，默认为9092，是与broker映射的端口
[qiyou@hadoop101 config]$ vim server.properties 

分别为0，1，2

更改临时目录

更改链接

修改java环境变量

【可以不卸载】将/usr/local/soft下的jdk移动到/opt/module中

mv jdk1.8.0_321/ /opt/module/
vim /etc/profile
source /etc/profile
使用java -version检查

下载hadoop

下载到目录【https://archive.apache.org/dist/hadoop/common/hadoop-3.1.3/】

[root@hadoop101 jdk1.8.0_321] cd /opt/software/
[root@hadoop101 software] ll
总用量 414616
-rw-r--r--. 1 root root 338075860 4月  21 15:26 hadoop-3.1.3.tar.gz
-rw-r--r--. 1 root root  86486610 4月  21 14:27 kafka_2.12-3.0.0.tgz

解压到/opt/module下

tar -zxvf hadoop-3.1.3.tar.gz -C /opt/module/

[root@hadoop101 module] ll
总用量 0
drwxr-xr-x. 9 qiyou qiyou 149 9月  12 2019 hadoop-3.1.3
drwxr-xr-x. 8 10143 10143 273 12月 16 03:30 jdk1.8.0_321
drwxr-xr-x. 7 qiyou qiyou 105 9月   9 2021 kafka

进入hadoop下

[root@hadoop101 module] cd hadoop-3.1.3/
[root@hadoop101 hadoop-3.1.3] ll
总用量 176
drwxr-xr-x. 2 qiyou qiyou    183 9月  12 2019 bin
drwxr-xr-x. 3 qiyou qiyou     20 9月  12 2019 etc
drwxr-xr-x. 2 qiyou qiyou    106 9月  12 2019 include
drwxr-xr-x. 3 qiyou qiyou     20 9月  12 2019 lib
drwxr-xr-x. 4 qiyou qiyou    288 9月  12 2019 libexec
-rw-rw-r--. 1 qiyou qiyou 147145 9月   4 2019 LICENSE.txt
-rw-rw-r--. 1 qiyou qiyou  21867 9月   4 2019 NOTICE.txt
-rw-rw-r--. 1 qiyou qiyou   1366 9月   4 2019 README.txt
drwxr-xr-x. 3 qiyou qiyou   4096 9月  12 2019 sbin
drwxr-xr-x. 4 qiyou qiyou     31 9月  12 2019 share

添加hadoop环境变量

vim /etc/profile

source /etc/profile

检查是否配置成功

hadoop version

Hadoop 3.1.3
Source code repository https://gitbox.apache.org/repos/asf/hadoop.git -r ba631c436b806728f8ec2f54ab1e289526c90579
Compiled by ztang on 2019-09-12T02:47Z
Compiled with protoc 2.5.0
From source with checksum ec785077c385118ac91aadde5ec9799
This command was run using /opt/module/hadoop-3.1.3/share/hadoop/common/hadoop-common-3.1.3.jar

【如果命令不可用，建议重启】

bin 目录：存放对 Hadoop 相关服务（hdfs，yarn，mapred）进行操作的脚本
etc 目录：Hadoop 的配置文件目录，存放 Hadoop 的配置文件
lib 目录：存放 Hadoop 的本地库（对数据进行压缩解压缩功能）
sbin 目录：存放启动或停止 Hadoop 相关服务的脚本
share 目录：存放 Hadoop 的依赖 jar 包、文档、和官方案例

SSH登录

cd .ssh

[root@hadoop103 .ssh] cat known_hosts

localhost ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKhHnCkQ7ILBCsYCBx/TrRztQBvnrr2SBX5EztvMOvmyRgJcf0eGnHFxQEk4ozQPpqN28f/HXtj8HV8mYWQJuF8=

#登出
[root@hadoop103 .ssh] exit
登出
Connection to hadoop103 closed.

rsync 远程同步工具

rsync 主要用于备份和镜像。具有速度快、避免复制相同内容和支持符号链接的优点。rsync 和 scp 区别：用 rsync 做文件的复制要比 scp 的速度快，rsync 只对差异文件做更新。scp 是把所有文件都复制过去。

注意事项！！

启动kafka之前必须先启动zookeeper，否则启动不成功

[root@hadoop101 kafka] bin/kafka-server-start.sh -daemon config/server.properties 
#无响应

解压到/opt/module

[root@hadoop101 module] ll
总用量 0
drwxr-xr-x. 6 root  root  133 4月  21 17:07 apache-zookeeper-3.6.3-bin
drwxr-xr-x. 9 qiyou qiyou 149 9月  12 2019 hadoop-3.1.3
drwxr-xr-x. 8 qiyou qiyou 273 12月 16 03:30 jdk1.8.0_321
drwxr-xr-x. 8 qiyou qiyou 117 4月  21 16:40 kafka

查看信息

[root@hadoop101 module] cd zookeeper-3.6.3/
[root@hadoop101 zookeeper-3.6.3] ll
总用量 36
drwxr-xr-x. 2 qiyou qiyou  4096 4月   9 2021 bin
drwxr-xr-x. 2 qiyou qiyou    77 4月   9 2021 conf
drwxr-xr-x. 5 qiyou qiyou  4096 4月   9 2021 docs
drwxr-xr-x. 2 root  root   4096 4月  21 17:07 lib
-rw-r--r--. 1 qiyou qiyou 11358 4月   9 2021 LICENSE.txt
-rw-r--r--. 1 qiyou qiyou   432 4月   9 2021 NOTICE.txt
-rw-r--r--. 1 qiyou qiyou  1963 4月   9 2021 README.md
-rw-r--r--. 1 qiyou qiyou  3166 4月   9 2021 README_packaging.md
[root@hadoop101 zookeeper-3.6.3] cd bin/
[root@hadoop101 bin] ll
总用量 64
-rwxr-xr-x. 1 qiyou qiyou   232 4月   9 2021 README.txt
-rwxr-xr-x. 1 qiyou qiyou  2066 4月   9 2021 zkCleanup.sh
-rwxr-xr-x. 1 qiyou qiyou  1158 4月   9 2021 zkCli.cmd
-rwxr-xr-x. 1 qiyou qiyou  1620 4月   9 2021 zkCli.sh
-rwxr-xr-x. 1 qiyou qiyou  1843 4月   9 2021 zkEnv.cmd
-rwxr-xr-x. 1 qiyou qiyou  3690 4月   9 2021 zkEnv.sh
-rwxr-xr-x. 1 qiyou qiyou  1286 4月   9 2021 zkServer.cmd
-rwxr-xr-x. 1 qiyou qiyou  4559 4月   9 2021 zkServer-initialize.sh
-rwxr-xr-x. 1 qiyou qiyou 11332 4月   9 2021 zkServer.sh
-rwxr-xr-x. 1 qiyou qiyou   988 4月   9 2021 zkSnapShotToolkit.cmd
-rwxr-xr-x. 1 qiyou qiyou  1377 4月   9 2021 zkSnapShotToolkit.sh
-rwxr-xr-x. 1 qiyou qiyou   996 4月   9 2021 zkTxnLogToolkit.cmd
-rwxr-xr-x. 1 qiyou qiyou  1385 4月   9 2021 zkTxnLogToolkit.sh

创建zkData，修改配置文件的目录

[root@hadoop101 zookeeper-3.6.3] mkdir zkData
[root@hadoop101 zookeeper-3.6.3] cd zkData/
[root@hadoop101 zkData] pwd
/opt/module/zookeeper-3.6.3/zkData
[root@hadoop101 zkData] cd ..
[root@hadoop101 zookeeper-3.6.3] cd conf/
[root@hadoop101 conf] ll
总用量 12
-rw-r--r--. 1 qiyou qiyou  535 4月   9 2021 configuration.xsl
-rw-r--r--. 1 qiyou qiyou 3435 4月   9 2021 log4j.properties
-rw-r--r--. 1 qiyou qiyou 1148 4月  21 20:05 zoo.cfg

修改zoo.cfg中datadir目录位置，添加集群对应信息

[root@hadoop101 conf] vim zoo.cfg 

#1、dataDir=/opt/module/zookeeper-3.6.3/zkData

#2、server.1=hadoop101:2888:3888
#   erver.2=hadoop102:2888:3888
#   server.3=hadoop103:2888:3888

启动zookeeper成功

[root@hadoop101 zookeeper-3.6.3] bin/zkServer.sh start
ZooKeeper JMX enabled by default
Using config: /opt/module/zookeeper-3.6.3/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED

编写zk.sh

#!/bin/bash
case $1 in
"start"){
        for i in hadoop101 hadoop102 hadoop103
        do
                echo ---------- zookeeper $i 启动 ------------
                ssh $i "/opt/module/zookeeper-3.6.3/bin/zkServer.sh start"
        done
};;
"stop"){
        for i in hadoop101 hadoop102 hadoop103
        do
                echo ---------- zookeeper $i 停止 ------------
                ssh $i "/opt/module/zookeeper-3.6.3/bin/zkServer.sh stop"
        done
};;
"status"){
        for i in hadoop101 hadoop102 hadoop103
        do
                echo ---------- zookeeper $i 状态 ------------
                ssh $i "/opt/module/zookeeper-3.6.3/bin/zkServer.sh status"
        done
};;
esac

启动脚本成功！

[root@hadoop101 bin] ./zk.sh start
---------- zookeeper hadoop101 启动 ------------
root@hadoop101's password: 
/usr/bin/java
ZooKeeper JMX enabled by default
Using config: /opt/module/zookeeper-3.6.3/bin/../conf/zoo.cfg
Starting zookeeper ... already running as process 80413.
---------- zookeeper hadoop102 启动 ------------
root@hadoop102's password: 
/usr/bin/java
ZooKeeper JMX enabled by default
Using config: /opt/module/zookeeper-3.6.3/bin/../conf/zoo.cfg
Starting zookeeper ... already running as process 74734.
---------- zookeeper hadoop103 启动 ------------
root@hadoop103's password: 
/usr/bin/java
ZooKeeper JMX enabled by default
Using config: /opt/module/zookeeper-3.6.3/bin/../conf/zoo.cfg
Starting zookeeper ... already running as process 75013.

利用zk.sh脚本启动集群，查看状态

[root@hadoop101 bin] ./zk.sh status
---------- zookeeper hadoop101 状态 ------------
root@hadoop101's password: 
/usr/bin/java
ZooKeeper JMX enabled by default
Using config: /opt/module/zookeeper-3.6.3/bin/../conf/zoo.cfg
Client port found: 2181. Client address: localhost. Client SSL: false.
#表示hadoop101作为leader
Mode: leader
---------- zookeeper hadoop102 状态 ------------
root@hadoop102's password: 
/usr/bin/java
ZooKeeper JMX enabled by default
Using config: /opt/module/zookeeper-3.6.3/bin/../conf/zoo.cfg
Client port found: 2181. Client address: localhost. Client SSL: false.
#hadoop102为其中一个follower
Mode: follower
---------- zookeeper hadoop103 状态 ------------
root@hadoop103's password: 
/usr/bin/java
ZooKeeper JMX enabled by default
Using config: /opt/module/zookeeper-3.6.3/bin/../conf/zoo.cfg
Client port found: 2181. Client address: localhost. Client SSL: false.
#hadoop103为另一个follower
Mode: follower

测试xsync脚本

在根目录下创建bin目录，在bin目录创建xsync脚本

#!/bin/bash 

#1. 判断参数个数 
# 判断参数是否小于1
if [ $# -lt 1 ]
then
    echo Not Enough Arguement! 
    exit;
fi

#2. 遍历集群所有机器 
# 对101,102,103都进行分发
for host in hadoop101 hadoop102 hadoop103
do
   echo ====================  $host  ==================== 
   #3. 遍历所有目录，挨个发送 
   for file in $@
   do

        #4. 判断文件是否存在 
        if [ -e $file ]
            then
                #5. 获取父目录 
                pdir=$(cd -P $(dirname $file); pwd)

                #6. 获取当前文件的名称 
                fname=$(basename $file)
                ssh $host "mkdir -p $pdir"
                rsync -av $pdir/$fname $host:$pdir
            # 如果不存在
            else
                echo $file does not exists! 
        fi
    done
done

给xsync赋权，并启动测试

[root@hadoop101 bin] chmod +x xsync
[root@hadoop101 bin] cd ..
[root@hadoop101 ~] xsync bin/
==================== hadoop101 ====================
root@hadoop101's password: 
root@hadoop101's password: 
sending incremental file list

sent 76 bytes  received 17 bytes  26.57 bytes/sec
total size is 872  speedup is 9.38
==================== hadoop102 ====================
root@hadoop102's password: 
root@hadoop102's password: 
sending incremental file list
bin/
bin/xsync

sent 822 bytes  received 51 bytes  249.43 bytes/sec
total size is 872  speedup is 1.00
==================== hadoop103 ====================
root@hadoop103's password: 
root@hadoop103's password:
sending incremental file list
bin/
bin/xsync

sent 822 bytes  received 51 bytes  83.14 bytes/sec
total size is 872  speedup is 1.00

同步到其他集群成功

$[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-czZOX7di-1650636241089)(C:\Users\QIYOU\AppData\Roaming\Typora\typora-user-images\image-20220422144016617.png)]$

分发kafka

[root@hadoop101 module] xsync kafka/
==================== hadoop101 ====================
root@hadoop101's password: 
root@hadoop101's password: 
sending incremental file list

sent 6,131 bytes  received 25 bytes  1,368.00 bytes/sec
total size is 90,301,516  speedup is 14,668.86
==================== hadoop102 ====================
root@hadoop102's password: 
root@hadoop102's password: 
sending incremental file list
kafka/
kafka/bin/
kafka/bin/windows/
kafka/config/
kafka/config/server.properties
kafka/config/kraft/
kafka/libs/
kafka/licenses/
kafka/logs/
kafka/logs/controller.log
kafka/logs/kafka-authorizer.log
kafka/logs/kafka-request.log
kafka/logs/kafkaServer-gc.log.0.current
kafka/logs/kafkaServer.out
kafka/logs/log-cleaner.log
kafka/logs/server.log
kafka/logs/state-change.log
kafka/site-docs/

sent 150,452 bytes  received 869 bytes  43,234.57 bytes/sec
total size is 90,301,516  speedup is 596.75
==================== hadoop103 ====================
root@hadoop103's password: 
root@hadoop103's password: 
sending incremental file list
kafka/
kafka/bin/
kafka/bin/windows/
kafka/config/
kafka/config/server.properties
kafka/config/kraft/
kafka/libs/
kafka/licenses/
kafka/logs/
kafka/logs/controller.log
kafka/logs/kafka-authorizer.log
kafka/logs/kafka-request.log
kafka/logs/kafkaServer-gc.log.0.current
kafka/logs/kafkaServer.out
kafka/logs/log-cleaner.log
kafka/logs/server.log
kafka/logs/state-change.log
kafka/site-docs/

sent 150,452 bytes  received 869 bytes  43,234.57 bytes/sec
total size is 90,301,516  speedup is 596.75

topic命令操作

增删改查主题，设置分区数、分区副本

增

创建主题

[root@hadoop101 kafka] bin/kafka-topics.sh --bootstrap-server hadoop101:2181 --topic first --create --partitions 1 --replication 3
#出现错误
Exception in thread "main" joptsimple.UnrecognizedOptionException: replication is not a recognized option
	at joptsimple.OptionException.unrecognizedOption(OptionException.java:108)
	at joptsimple.OptionParser.handleLongOptionToken(OptionParser.java:510)
	at joptsimple.OptionParserState$2.handleArgument(OptionParserState.java:56)
	at joptsimple.OptionParser.parse(OptionParser.java:396)
	at kafka.admin.TopicCommand$TopicCommandOptions.<init>(TopicCommand.scala:517)
	at kafka.admin.TopicCommand$.main(TopicCommand.scala:47)
	at kafka.admin.TopicCommand.main(TopicCommand.scala)
	
#敲错代码，修改
[root@hadoop101 kafka] bin/kafka-topics.sh --bootstrap-server hadoop101:2181 --topic first --create --partitions 1 --replication-factor 3

#超时
[2022-04-22 15:44:52,762] ERROR org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment. Call: createTopics
 (kafka.admin.TopicCommand$)
 
 #错误原因，监听端口默认为9092，不是2181
 [root@hadoop101 kafka] bin/kafka-topics.sh --bootstrap-server hadoop101:9092 --topic first --create --partitions 1 --replication-factor 3
Created topic first.

#查看具体主题信息
[root@hadoop101 kafka] bin/kafka-topics.sh --bootstrap-server hadoop101:9092 --list
first
[root@hadoop101 kafka] bin/kafka-topics.sh --bootstrap-server hadoop101:9092 --topic first --describe
Topic: first	TopicId: ICmWRs1NS5e_fvDg3le6JA	PartitionCount: 1	ReplicationFactor: 3	Configs: segment.bytes=1073741824
	Topic: first	Partition: 0	Leader: 1	Replicas: 1,0,2	Isr: 1,0,2