Kafak

1. 安装部署

[atguigu@hadoop102 software]$ tar -zxvf kafka_2.13-3.3.1.tgz -C /opt/module/
[atguigu@hadoop102 module]$ cd kafka_2.13-3.3.1/
[atguigu@hadoop102 kafka_2.13-3.3.1]$ ll
total 64
drwxr-xr-x. 3 atguigu atguigu  4096 Sep 30 03:06 bin
drwxr-xr-x. 3 atguigu atguigu  4096 Sep 30 03:06 config
drwxr-xr-x. 2 atguigu atguigu  8192 Nov 29 10:41 libs
-rw-rw-r--. 1 atguigu atguigu 14842 Sep 30 03:03 LICENSE
drwxr-xr-x. 2 atguigu atguigu   284 Sep 30 03:06 licenses
-rw-rw-r--. 1 atguigu atguigu 28184 Sep 30 03:03 NOTICE
drwxr-xr-x. 2 atguigu atguigu    44 Sep 30 03:06 site-docs
[atguigu@hadoop102 kafka_2.13-3.3.1]$ vim config/server.properties 

# 输入以下内容
#broker 的全局唯一编号,不能重复,只能是数字
broker.id=0
#kafka 运行日志(数据)存放的路径，路径不需要提前创建，kafka 自动帮你创建，可以配置多个磁盘路径，路径与路径之间可以用"，"分隔
log.dirs=/opt/module/kafka/datas
#配置连接 Zookeeper 集群地址（在 zk 根目录下创建/kafka，方便管理）
zookeeper.connect=hadoop102:2181,hadoop103:2181,hadoop104:2181/kafka

[atguigu@hadoop102 module]$ xsync kafka_2.13-3.3.1/

#分别在 hadoop103 和 hadoop104 上修改配置文件/opt/module/kafka/config/server.properties中的 broker.id=1、broker.id=2
# 注：broker.id 不得重复，整个集群中唯一。

[atguigu@hadoop102 module]$ sudo vim /etc/profile.d/my_env.sh

# 增加一下内容
#KAFKA_HOME
export KAFKA_HOME=/opt/module/kafka_2.13-3.3.1
export PATH=$PATH:$KAFKA_HOME/bin

[atguigu@hadoop102 module]$ source /etc/profile
[atguigu@hadoop102 module]$ sudo /home/atguigu/bin/xsync /etc/profile.d/my_env.sh 
[atguigu@hadoop102 module]$ cd /home/atguigu/bin/

# 集群启动停止脚本
[atguigu@hadoop102 bin]$ vim kf.sh
[atguigu@hadoop102 bin]$ chmod +x kf.sh
[atguigu@hadoop102 bin]$ ll

注意： 停止 Kafka 集群时，一定要等 Kafka 所有节点进程全部停止后再停止 Zookeeper集群。因为 Zookeeper 集群当中记录着 Kafka 集群相关信息，Zookeeper 集群一旦先停止，Kafka 集群就没有办法再获取停止进程的信息，只能手动杀死 Kafka 进程了。

2. Topic命令

2.1 主题（Topic）命令操作

# 创建topic
[atguigu@hadoop102 kafka_2.13-3.3.1]$ bin/kafka-topics.sh --bootstrap-server hadoop102:9092 --create --partitions 1 --replication-factor 3 --topic first
OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N
Created topic first.
# 查看topic
[atguigu@hadoop102 kafka_2.13-3.3.1]$ bin/kafka-topics.sh --bootstrap-server hadoop102:9092 --list
OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N
first
# 详情查看
[atguigu@hadoop102 kafka_2.13-3.3.1]$ bin/kafka-topics.sh --bootstrap-server hadoop102:9092 --describe --topic first
OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N
Topic: first	TopicId: haUX9KCST-O6c2JbgsYjDQ	PartitionCount: 1	ReplicationFactor: 3	Configs: 
	Topic: first	Partition: 0	Leader: 1	Replicas: 1,2,3	Isr: 1,2,3
# 修改分区数
[atguigu@hadoop102 kafka_2.13-3.3.1]$ bin/kafka-topics.sh --bootstrap-server hadoop102:9092 --alter --topic first --partitions 3
OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N
[atguigu@hadoop102 kafka_2.13-3.3.1]$ bin/kafka-topics.sh --bootstrap-server hadoop102:9092 --describe --topic first
OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N
Topic: first	TopicId: haUX9KCST-O6c2JbgsYjDQ	PartitionCount: 3	ReplicationFactor: 3	Configs: 
	Topic: first	Partition: 0	Leader: 1	Replicas: 1,2,3	Isr: 1,2,3
	Topic: first	Partition: 1	Leader: 2	Replicas: 2,3,1	Isr: 2,3,1
	Topic: first	Partition: 2	Leader: 3	Replicas: 3,1,2	Isr: 3,1,2
# 删除topic
[atguigu@hadoop102 kafka_2.13-3.3.1]$ bin/kafka-topics.sh --bootstrap-server hadoop102:9092 --delete --topic first
OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N

2.2 生产者（Producer）命令操作

[atguigu@hadoop102 kafka_2.13-3.3.1]$ bin/kafka-console-producer.sh --bootstrap-server hadoop102:9092 --topic first
OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N
>hello
>
>

2.3 消费者（Consumer）命令操作

[atguigu@hadoop102 kafka_2.13-3.3.1]$ bin/kafka-console-consumer.sh --bootstrap-server hadoop102:9092 --topic first
OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N
hello

# 把主题中所有的数据都读取出来(包括历史数据)
[atguigu@hadoop102 kafka]$ bin/kafka-console-consumer.sh --bootstrap-server hadoop102:9092 --from-beginning --topic first

3. Kafak生产者

3.1 生产者消息发送流程

3.1.1 发送原理

3.2 异步发送API

3.2.1 普通异步发送