1. 安装部署
[atguigu@hadoop102 software]$ tar -zxvf kafka_2.13-3.3.1.tgz -C /opt/module/
[atguigu@hadoop102 module]$ cd kafka_2.13-3.3.1/
[atguigu@hadoop102 kafka_2.13-3.3.1]$ ll
total 64
drwxr-xr-x. 3 atguigu atguigu 4096 Sep 30 03:06 bin
drwxr-xr-x. 3 atguigu atguigu 4096 Sep 30 03:06 config
drwxr-xr-x. 2 atguigu atguigu 8192 Nov 29 10:41 libs
-rw-rw-r--. 1 atguigu atguigu 14842 Sep 30 03:03 LICENSE
drwxr-xr-x. 2 atguigu atguigu 284 Sep 30 03:06 licenses
-rw-rw-r--. 1 atguigu atguigu 28184 Sep 30 03:03 NOTICE
drwxr-xr-x. 2 atguigu atguigu 44 Sep 30 03:06 site-docs
[atguigu@hadoop102 kafka_2.13-3.3.1]$ vim config/server.properties
# 输入以下内容
#broker 的全局唯一编号,不能重复,只能是数字
broker.id=0
#kafka 运行日志(数据)存放的路径,路径不需要提前创建,kafka 自动帮你创建,可以配置多个磁盘路径,路径与路径之间可以用","分隔
log.dirs=/opt/module/kafka/datas
#配置连接 Zookeeper 集群地址(在 zk 根目录下创建/kafka,方便管理)
zookeeper.connect=hadoop102:2181,hadoop103:2181,hadoop104:2181/kafka
[atguigu@hadoop102 module]$ xsync kafka_2.13-3.3.1/
#分别在 hadoop103 和 hadoop104 上修改配置文件/opt/module/kafka/config/server.properties中的 broker.id=1、broker.id=2
# 注:broker.id 不得重复,整个集群中唯一。
[atguigu@hadoop102 module]$ sudo vim /etc/profile.d/my_env.sh
# 增加一下内容
#KAFKA_HOME
export KAFKA_HOME=/opt/module/kafka_2.13-3.3.1
export PATH=$PATH:$KAFKA_HOME/bin
[atguigu@hadoop102 module]$ source /etc/profile
[atguigu@hadoop102 module]$ sudo /home/atguigu/bin/xsync /etc/profile.d/my_env.sh
[atguigu@hadoop102 module]$ cd /home/atguigu/bin/
# 集群启动停止脚本
[atguigu@hadoop102 bin]$ vim kf.sh
[atguigu@hadoop102 bin]$ chmod +x kf.sh
[atguigu@hadoop102 bin]$ ll
注意: 停止 Kafka
集群时,一定要等 Kafka
所有节点进程全部停止后再停止 Zookeeper
集群。因为 Zookeeper
集群当中记录着 Kafka
集群相关信息,Zookeeper
集群一旦先停止,Kafka 集群就没有办法再获取停止进程的信息,只能手动杀死 Kafka 进程了。
2. Topic命令
2.1 主题(Topic)命令操作
# 创建topic
[atguigu@hadoop102 kafka_2.13-3.3.1]$ bin/kafka-topics.sh --bootstrap-server hadoop102:9092 --create --partitions 1 --replication-factor 3 --topic first
OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N
Created topic first.
# 查看topic
[atguigu@hadoop102 kafka_2.13-3.3.1]$ bin/kafka-topics.sh --bootstrap-server hadoop102:9092 --list
OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N
first
# 详情查看
[atguigu@hadoop102 kafka_2.13-3.3.1]$ bin/kafka-topics.sh --bootstrap-server hadoop102:9092 --describe --topic first
OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N
Topic: first TopicId: haUX9KCST-O6c2JbgsYjDQ PartitionCount: 1 ReplicationFactor: 3 Configs:
Topic: first Partition: 0 Leader: 1 Replicas: 1,2,3 Isr: 1,2,3
# 修改分区数
[atguigu@hadoop102 kafka_2.13-3.3.1]$ bin/kafka-topics.sh --bootstrap-server hadoop102:9092 --alter --topic first --partitions 3
OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N
[atguigu@hadoop102 kafka_2.13-3.3.1]$ bin/kafka-topics.sh --bootstrap-server hadoop102:9092 --describe --topic first
OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N
Topic: first TopicId: haUX9KCST-O6c2JbgsYjDQ PartitionCount: 3 ReplicationFactor: 3 Configs:
Topic: first Partition: 0 Leader: 1 Replicas: 1,2,3 Isr: 1,2,3
Topic: first Partition: 1 Leader: 2 Replicas: 2,3,1 Isr: 2,3,1
Topic: first Partition: 2 Leader: 3 Replicas: 3,1,2 Isr: 3,1,2
# 删除topic
[atguigu@hadoop102 kafka_2.13-3.3.1]$ bin/kafka-topics.sh --bootstrap-server hadoop102:9092 --delete --topic first
OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N
2.2 生产者(Producer)命令操作
[atguigu@hadoop102 kafka_2.13-3.3.1]$ bin/kafka-console-producer.sh --bootstrap-server hadoop102:9092 --topic first
OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N
>hello
>
>
2.3 消费者(Consumer)命令操作
[atguigu@hadoop102 kafka_2.13-3.3.1]$ bin/kafka-console-consumer.sh --bootstrap-server hadoop102:9092 --topic first
OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N
hello
# 把主题中所有的数据都读取出来(包括历史数据)
[atguigu@hadoop102 kafka]$ bin/kafka-console-consumer.sh --bootstrap-server hadoop102:9092 --from-beginning --topic first