kafka 操作命令
在 0.9.0.0 之后的 Kafka，出现了几个新变动，一个是在 Server 端增加了 GroupCoordinator 这个角色，另一个较大的变动是将 topic 的 offset 信息由之前存储在 zookeeper 上改为存储到一个特殊的 topic(__consumer_offsets)中。
启动 Kafka
# 启动zk集群
zkServer.sh start
# 启动kafka集群
kafka-server-start.sh -daemon /opt/kafka_2.13-3.2.0/config/server.properties
# 停止kafka
kafka-server-stop.sh
topic
[root@lsyk02 ~]# kafka-topics.sh
Create, delete, describe, or change a topic.
Option Description
------ -----------
--alter Alter the number of partitions,
replica assignment, and/or
configuration for the topic.
--at-min-isr-partitions if set when describing topics, only
show partitions whose isr count is
equal to the configured minimum.
--bootstrap-server <String: server to REQUIRED: The Kafka server to connect
connect to> to.
--command-config <String: command Property file containing configs to be
config property file> passed to Admin Client. This is used
only with --bootstrap-server option
for describing and altering broker
configs.
--config <String: name=value> A topic configuration override for the
topic being created or altered. The
following is a list of valid
configurations:
cleanup.policy
compression.type
delete.retention.ms
file.delete.delay.ms
flush.messages
flush.ms
follower.replication.throttled.
replicas
index.interval.bytes
leader.replication.throttled.replicas
local.retention.bytes
local.retention.ms
max.compaction.lag.ms
max.message.bytes
message.downconversion.enable
message.format.version
message.timestamp.difference.max.ms
message.timestamp.type
min.cleanable.dirty.ratio
min.compaction.lag.ms
min.insync.replicas
preallocate
remote.storage.enable
retention.bytes
retention.ms
segment.bytes
segment.index.bytes
segment.jitter.ms
segment.ms
unclean.leader.election.enable
See the Kafka documentation for full
details on the topic configs. It is
supported only in combination with --
create if --bootstrap-server option
is used (the kafka-configs CLI
supports altering topic configs with
a --bootstrap-server option).
--create Create a new topic.
--delete Delete a topic
--delete-config <String: name> A topic configuration override to be
removed for an existing topic (see
the list of configurations under the
--config option). Not supported with
the --bootstrap-server option.
--describe List details for the given topics.
--disable-rack-aware Disable rack aware replica assignment
--exclude-internal exclude internal topics when running
list or describe command. The
internal topics will be listed by
default
--help Print usage information.
--if-exists if set when altering or deleting or
describing topics, the action will
only execute if the topic exists.
--if-not-exists if set when creating topics, the
action will only execute if the
topic does not already exist.
--list List all available topics.
--partitions <Integer: # of partitions> The number of partitions for the topic
being created or altered (WARNING:
If partitions are increased for a
topic that has a key, the partition
logic or ordering of the messages
will be affected). If not supplied
for create, defaults to the cluster
default.
--replica-assignment <String: A list of manual partition-to-broker
broker_id_for_part1_replica1 : assignments for the topic being
broker_id_for_part1_replica2 , created or altered.
broker_id_for_part2_replica1 :
broker_id_for_part2_replica2 , ...>
--replication-factor <Integer: The replication factor for each
replication factor> partition in the topic being
created. If not supplied, defaults
to the cluster default.
--topic <String: topic> The topic to create, alter, describe
or delete. It also accepts a regular
expression, except for --create
option. Put topic name in double
quotes and use the '\' prefix to
escape regular expression symbols; e.
g. "test\.topic".
--topic-id <String: topic-id> The topic-id to describe.This is used
only with --bootstrap-server option
for describing topics.
--topics-with-overrides if set when describing topics, only
show topics that have overridden
configs
--unavailable-partitions if set when describing topics, only
show partitions whose leader is not
available
--under-min-isr-partitions if set when describing topics, only
show partitions whose isr count is
less than the configured minimum.
--under-replicated-partitions if set when describing topics, only
show under replicated partitions
--version Display Kafka version.
-
创建topic
kafka-topics.sh --create --bootstrap-server lsyk01:9092,lsyk02:9092,lsyk03:9092 --replication-factor 1 --partitions 3 --topic fazi_01--topic指定 Topic 名--partitions指定分区数--replication-factor指定副本数–bootstrap-server 制定配置的kafka，不是kafka的端口，是在server.propertis里配置的,这里可以只写1个
-
列出topic
kafka-topics.sh --list --bootstrap-server lsyk01:9092,lsyk02:9092,lsyk03:9092 -
查看指定的topic
kafka-topics.sh --describe --bootstrap-server lsyk01:9092 --topic fazi_01
Partition:0 表示该分区的id为0
leader: 9 表示分区的首领副本所在的broker（本例子中broker.id配置为9,所以这里显示9,具体在config/server.properties配置。这里只有一个分区，所以首领分区也就是自己）
Replicas: 9 表示分区的跟随副本所在的broker
Isr: 9 表示分区的同步副本所在的broker(同步副本可以认为跟首领副本准实时同步的副本，可以配置判断条件，后面会讲，首领副本挂掉后，服务器会从同步副本中选举新的首领) -
改变kafka的分区数,只能增大不能缩小
kafka-topics.sh --bootstrap-server lsyk01:9092 --alter --topic fazi_01 --partitions 5
-
指定分区的offset最大和最小值（topic当前的offset情况，及时没有被消费，offset也一直+1）
kafka-run-class.sh kafka.tools.GetOffsetShell --topic fazi_01 --time -1 --broker-list lsyk01:9092其中，time 为 -1 时表示最大值，为 -2 时表示最小值：
-
删除topic
kafka-topics.sh --bootstrap-server lsyk02:9092 --topic fazi_02 --delete -
消费消息
# 从头开始 kafka-console-consumer.sh --bootstrap-server lsyk02:9092 --topic fazi_01 --from-beginning # 从尾部开始，必须指定分区 kafka-console-consumer.sh --bootstrap-server lsyk02:9092 --topic fazi_01 --offset latest --partition 0 # 从尾部开始，必须指定分区，用来查看topic里的某个分区的最近：N条记录，即offset的开始点为latest - N kafka-console-consumer.sh --bootstrap-server lsyk02:9092 --topic fazi_01 --offset 10 --partition 0
消费者group
-
消费者group列表
kafka-consumer-groups.sh --bootstrap-server lsyk01:9092 --list -
查看Group详情[用来判断是否有延迟数据]
kafka-consumer-groups.sh --bootstrap-server lsyk01:9092 --group test_group --describe
-
指定group,如果没有group则会创建
kafka-console-consumer.sh --bootstrap-server lsyk01:9092 --topic fazi_01 -group test_group --from-beginning -
删除group
kafka-consumer-groups.sh --bootstrap-server lsyk01:9092 --group test_group --delete注意：
kafka重启，消息不会丢失，
生产者接着生产，消息会一直保存在topic
查看消费者描述的时候，会看到消息有积压
消费者开启时候，即使写了–from-beginning,也是从上次消费的offset开始，没数据也没关系，很快就会把offset更新到系统。
生产者，只负责生产，数据的格式一概不管，原样保存
如果将主题指定到新的消费者，则会新的消费者根据指定的方式继续续消费
一个用户组的描述，是把该用户消费的所有的topic信息都列举出来了，互不干扰
kafka修改topic保留时间
kafka-topics.sh修改配置已被弃用，后续版本会删除，建议需要统一使用kafka-configs.sh来修改参数。
热生效，且永久生效。
kafka-configs.sh --describe --bootstrap-server lsyk01:9092 --entity-type topics --entity-name fazi_01
```shell
#修改默认的消息保留的时间：默认是7243600*1000
kafka-configs.sh --bootstrap-server lsyk01:9092 --alter --entity-name fazi_01 --entity-type topics --add-config retention.ms=30000
```
配置完成后，所有超过时间的数据会先被标记为deleted，稍后清空全部标记为deleted的文件 .
查看/opt/kafka_2.13-3.2.0/logs/server.log会发现很多删除了：
191
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
