kafka使用（一）--- 初步使用

目录

1.环境搭建

2.常用命令

3.消费者与分区数

4.java程序中订阅topic刷新

---

1.环境搭建

# 下载安装包
wget https://archive.apache.org/dist/kafka/2.8.1/kafka_2.13-2.8.1.tgz
# 解压
tar xzf kafka_2.13-2.8.1.tgz

2.常用命令

# 启动服务
nohup sh bin/zookeeper-server-start.sh config/zookeeper.properties >> zookeeper.out &
nohup sh bin/kafka-server-start.sh config/server.properties >> kafka.out &
# 创建topic
bin/kafka-topics.sh --create --replication-factor 1 --partitions 1 --topic firsttopic --bootstrap-server 192.168.137.100:9092
# 查看topic
bin/kafka-topics.sh --describe --topic firsttopic --bootstrap-server 192.168.137.100:9092
# 生产者发送消息
bin/kafka-console-producer.sh --topic newtest001 --bootstrap-server 192.168.137.100:9092
# 消费者接收消息
bin/kafka-console-consumer.sh --topic newtest001 --from-beginning --bootstrap-server 192.168.137.100:9092
# 消费者以指定消费者组接收消息，当消费者组不存在会自动创建
bin/kafka-console-consumer.sh --topic newtest001 --from-beginning --bootstrap-server 192.168.137.100:9092  --group testgroup
# 查看消费者组
bin/kafka-consumer-groups.sh --bootstrap-server 192.168.137.100:9092 --list 
bin/kafka-consumer-groups.sh --bootstrap-server 192.168.137.100:9092 --describe --group testgroup
# 删除消费者组，只有消费者未注册的消费者组才可以被删除（即使这个消费者组中有未被消费的消息也可以删除）
bin/kafka-consumer-groups.sh --bootstrap-server 192.168.137.100:9092 --delete --group testgroup

3.消费者与分区数

当多个消费者以同一个消费者组来消费topic（与消费者个数同样的分区数）时，默认是轮流接收到消息；如果是以不同的消费者组下来消费，则都会收到生产者的每条消息。

当topic的分区数为1个，然后消费者组内的消费者为多个的情况：后注册的消费者消费所有消息。

4.java程序中订阅topic刷新

启动java程序，启动时存在无法订阅成功要求的所有topic的情况，但是5分钟间隔后，会自动回收之前分配的订阅并重新订阅，此时基本是可以完全成功订阅所有topic的。

2021-11-19 10:52:24 [INFO][org.apache.kafka.clients.consumer.internals.AbstractCoordinator][info][341]-> [Consumer clientId=consumer-1, groupId=wuye] Successfully joined group with generation 55
2021-11-19 10:52:24 [INFO][org.apache.kafka.clients.consumer.internals.AbstractCoordinator][info][341]-> [Consumer clientId=consumer-2, groupId=wuye] Successfully joined group with generation 55
2021-11-19 10:52:24 [INFO][org.apache.kafka.clients.consumer.internals.AbstractCoordinator][info][341]-> [Consumer clientId=consumer-4, groupId=wuye] Successfully joined group with generation 55
2021-11-19 10:52:24 [INFO][org.apache.kafka.clients.consumer.internals.ConsumerCoordinator][info][341]-> [Consumer clientId=consumer-4, groupId=wuye] Setting newly assigned partitions [meterValidResult-0]
2021-11-19 10:52:24 [INFO][org.apache.kafka.clients.consumer.internals.ConsumerCoordinator][info][341]-> [Consumer clientId=consumer-1, groupId=wuye] Setting newly assigned partitions [paymentMakeResult-2, paymentMakeResult-1, paymentMakeResult-0]
2021-11-19 10:52:24 [INFO][org.apache.kafka.clients.consumer.internals.ConsumerCoordinator][info][341]-> [Consumer clientId=consumer-2, groupId=wuye] Setting newly assigned partitions [paymentStateSync-1, paymentStateSync-0, paymentStateSync-2]
2021-11-19 10:52:24 [INFO][org.springframework.kafka.listener.KafkaMessageListenerContainer][onPartitionsAssigned][348]-> partitions assigned: [meterValidResult-0]
2021-11-19 10:52:24 [INFO][org.springframework.kafka.listener.KafkaMessageListenerContainer][onPartitionsAssigned][348]-> partitions assigned: [paymentMakeResult-2, paymentMakeResult-1, paymentMakeResult-0]
2021-11-19 10:52:24 [INFO][org.springframework.kafka.listener.KafkaMessageListenerContainer][onPartitionsAssigned][348]-> partitions assigned: [paymentStateSync-1, paymentStateSync-0, paymentStateSync-2]
2021-11-19 10:57:27 [INFO][org.apache.kafka.clients.consumer.internals.AbstractCoordinator][info][341]-> [Consumer clientId=consumer-3, groupId=wuye] Group coordinator hw_kafka2:29092 (id: 2147483645 rack: null) is unavailable or invalid, will attempt rediscovery
2021-11-19 10:57:27 [INFO][org.apache.kafka.clients.consumer.internals.AbstractCoordinator][info][341]-> [Consumer clientId=consumer-3, groupId=wuye] Discovered group coordinator hw_kafka2:29092 (id: 2147483645 rack: null)
2021-11-19 10:57:27 [INFO][org.apache.kafka.clients.consumer.internals.AbstractCoordinator][info][336]-> [Consumer clientId=consumer-3, groupId=wuye] (Re-)joining group
2021-11-19 10:57:27 [INFO][org.apache.kafka.clients.consumer.internals.ConsumerCoordinator][info][341]-> [Consumer clientId=consumer-2, groupId=wuye] Revoking previously assigned partitions [paymentStateSync-1, paymentStateSync-0, paymentStateSync-2]
2021-11-19 10:57:27 [INFO][org.springframework.kafka.listener.KafkaMessageListenerContainer][onPartitionsRevoked][343]-> partitions revoked: [paymentStateSync-1, paymentStateSync-0, paymentStateSync-2]
2021-11-19 10:57:27 [INFO][org.apache.kafka.clients.consumer.internals.AbstractCoordinator][info][336]-> [Consumer clientId=consumer-2, groupId=wuye] (Re-)joining group
2021-11-19 10:57:27 [INFO][org.apache.kafka.clients.consumer.internals.ConsumerCoordinator][info][341]-> [Consumer clientId=consumer-1, groupId=wuye] Revoking previously assigned partitions [paymentMakeResult-2, paymentMakeResult-1, paymentMakeResult-0]
2021-11-19 10:57:27 [INFO][org.apache.kafka.clients.consumer.internals.ConsumerCoordinator][info][341]-> [Consumer clientId=consumer-4, groupId=wuye] Revoking previously assigned partitions [meterValidResult-0]
2021-11-19 10:57:27 [INFO][org.springframework.kafka.listener.KafkaMessageListenerContainer][onPartitionsRevoked][343]-> partitions revoked: [meterValidResult-0]
2021-11-19 10:57:27 [INFO][org.springframework.kafka.listener.KafkaMessageListenerContainer][onPartitionsRevoked][343]-> partitions revoked: [paymentMakeResult-2, paymentMakeResult-1, paymentMakeResult-0]
2021-11-19 10:57:27 [INFO][org.apache.kafka.clients.consumer.internals.AbstractCoordinator][info][336]-> [Consumer clientId=consumer-1, groupId=wuye] (Re-)joining group
2021-11-19 10:57:27 [INFO][org.apache.kafka.clients.consumer.internals.AbstractCoordinator][info][336]-> [Consumer clientId=consumer-4, groupId=wuye] (Re-)joining group
2021-11-19 10:57:47 [INFO][org.apache.kafka.clients.consumer.internals.AbstractCoordinator][info][336]-> [Consumer clientId=consumer-1, groupId=wuye] (Re-)joining group
2021-11-19 10:57:47 [INFO][org.apache.kafka.clients.consumer.internals.AbstractCoordinator][info][336]-> [Consumer clientId=consumer-3, groupId=wuye] (Re-)joining group
2021-11-19 10:57:47 [INFO][org.apache.kafka.clients.consumer.internals.AbstractCoordinator][info][336]-> [Consumer clientId=consumer-2, groupId=wuye] (Re-)joining group
2021-11-19 10:57:47 [INFO][org.apache.kafka.clients.consumer.internals.AbstractCoordinator][info][341]-> [Consumer clientId=consumer-3, groupId=wuye] Successfully joined group with generation 57
2021-11-19 10:57:47 [INFO][org.apache.kafka.clients.consumer.internals.AbstractCoordinator][info][341]-> [Consumer clientId=consumer-1, groupId=wuye] Successfully joined group with generation 57
2021-11-19 10:57:47 [INFO][org.apache.kafka.clients.consumer.internals.AbstractCoordinator][info][341]-> [Consumer clientId=consumer-2, groupId=wuye] Successfully joined group with generation 57
2021-11-19 10:57:47 [INFO][org.apache.kafka.clients.consumer.internals.ConsumerCoordinator][info][341]-> [Consumer clientId=consumer-2, groupId=wuye] Setting newly assigned partitions [paymentStateSync-1, paymentStateSync-0, paymentStateSync-2]
2021-11-19 10:57:47 [INFO][org.apache.kafka.clients.consumer.internals.ConsumerCoordinator][info][341]-> [Consumer clientId=consumer-1, groupId=wuye] Setting newly assigned partitions [paymentMakeResult-2, paymentMakeResult-1, paymentMakeResult-0]
2021-11-19 10:57:47 [INFO][org.apache.kafka.clients.consumer.internals.ConsumerCoordinator][info][341]-> [Consumer clientId=consumer-3, groupId=wuye] Setting newly assigned partitions [benchmarkCheckResult-0, benchmarkCheckResult-1, benchmarkCheckResult-2]
2021-11-19 10:57:47 [INFO][org.springframework.kafka.listener.KafkaMessageListenerContainer][onPartitionsAssigned][348]-> partitions assigned: [paymentMakeResult-2, paymentMakeResult-1, paymentMakeResult-0]
2021-11-19 10:57:47 [INFO][org.springframework.kafka.listener.KafkaMessageListenerContainer][onPartitionsAssigned][348]-> partitions assigned: [paymentStateSync-1, paymentStateSync-0, paymentStateSync-2]
2021-11-19 10:57:47 [INFO][org.springframework.kafka.listener.KafkaMessageListenerContainer][onPartitionsAssigned][348]-> partitions assigned: [benchmarkCheckResult-0, benchmarkCheckResult-1, benchmarkCheckResult-2]

如上程序中包含：meterValidResult、benchmarkCheckResult、paymentMakeResult、paymentStateSync，启动时只成功订阅了meterValidResult、paymentMakeResult、paymentStateSync，5分钟之后才全部订阅成功

参考：

Kafka分区与消费者个数之间的关系_我是明星的专栏-CSDN博客