Kafka常用命令

最新推荐文章于 2022-02-26 16:50:55 发布

jerrfy_w

最新推荐文章于 2022-02-26 16:50:55 发布

阅读量126

点赞数

分类专栏： Kafka

本文链接：https://blog.csdn.net/wzj_wp/article/details/108282315

版权

Kafka 专栏收录该内容

1 篇文章 0 订阅

订阅专栏

topic：主题
partition:每个partition是一个有序的队列

创建topic

./kafka-topics.sh \
--create \
--zookeeper localhost:2182,localhost:2183,localhost:2184/kafka \
--partitions 3 \
--replication-factor 3 \
--topic bigdata

查看topic列表

./kafka-topics.sh \
--list \
--zookeeper localhost:2182,localhost:2183,localhost:2184/kafka

删除topic

./kafka-topics.sh \
--delete \
--zookeeper localhost:2182,localhost:2183,localhost:2184/kafka \
--topic wzj

删除之后topic会被自动标记为marked for deletion

[wzj@hadoop001 bin]$ ./kafka-topics.sh \
> --list \
> --zookeeper localhost:2182,localhost:2183,localhost:2184/kafka
kafka_topic_test
test
wzj - marked for deletion
如果需要彻底删除，去zk中找到对应目录下的topic,被标记删除的会在 admin/delete_topics/文件下

[zk: localhost:2184(CONNECTED) 5] ls /kafka
[cluster, controller_epoch, controller, brokers, admin, isr_change_notification, consumers, log_dir_event_notification, latest_producer_id_block, config]
[zk: localhost:2184(CONNECTED) 6] ls /kafka/admin/delete_topics
[wzj]
[zk: localhost:2184(CONNECTED) 7] rmr /kafka/admin/delete_topics/wzj

至此，topic被彻底删除
删除topic参考博客：https://blog.csdn.net/belalds/article/details/80575751

查看指定topic

./kafka-topics.sh \
--describe \
--zookeeper localhost:2182,localhost:2183,localhost:2184/kafka \
--topic bigdata

[wzj@hadoop001 bin]$ ./kafka-topics.sh \
> --describe \
> --zookeeper localhost:2182,localhost:2183,localhost:2184/kafka \
> --topic bigdata
Topic:bigdata	PartitionCount:3	ReplicationFactor:3	Configs:
	Topic: bigdata	Partition: 0	Leader: 22	Replicas: 22,33,11	Isr: 22,33,11
	Topic: bigdata	Partition: 1	Leader: 33	Replicas: 33,11,22	Isr: 33,11,22
	Topic: bigdata	Partition: 2	Leader: 11	Replicas: 11,22,33	Isr: 11,22,33

检测kafka是否正常：

生产者：kafka-console-producer.sh
./kafka-console-producer.sh \
--broker-list hadoop001:9092,hadoop001:9093,hadoop001:9094 \
--topic bigdata

[wzj@hadoop001 bin]$ ./kafka-console-producer.sh \
> --broker-list hadoop001:9092,hadoop001:9093,hadoop001:9094 \
> --topic bigdata
>
>1
>2
>3
>4
>5
>6
>7
>8
>9

消费者：kafka-console-consumer.sh
./kafka-console-consumer.sh \
--bootstrap-server hadoop001:9092,hadoop001:9093,hadoop001:9094 \
--topic bigdata \
--from-beginning

[wzj@hadoop001 bin]$ ./kafka-console-consumer.sh \
> --bootstrap-server hadoop001:9092,hadoop001:9093,hadoop001:9094 \
> --topic bigdata \
> --from-beginning

1
2
3
4
5
6
7
8
9

消费者断掉重新启动会发现数据全局乱序，但是局部有序，这正是验证了每个partition是一个有序队列

partition1：1 4 7
partition2：2 5 8
partition3：3 6 9
[wzj@hadoop001 bin]$ 
[wzj@hadoop001 bin]$ ./kafka-console-consumer.sh --bootstrap-server hadoop001:9092,hadoop001:9093,hadoop001:9094 --topic bigdata --from-beginning
2
5
8

3
6
9
1
4
7

offset

https://cwiki.apache.org/confluence/display/KAFKA/System+Tools

https://cwiki.apache.org/confluence/display/KAFKA/Replication+tools

https://github.com/apache/kafka/tree/0.8/core/src/main/scala/kafka/tools

offset 消息标识ID

segment 分段
一个partition被切割成多个相同大小的segment。
xxx.log 数据
xxx.index offset和物理位移pos

每个partition都是由一系列有序的不可变的消息组成,消息是被追加到partition.每个partition中的每个消息message都是有一个连续的序列号来标识，用于 partition的唯一标识

/home/wzj/app/kafka1/bin/kafka-run-class.sh kafka.tools.DumpLogSegments \
--files 00000000000000000000.log \
--print-data-log \
> 0.txt


[wzj@hadoop001 bigdata-1]$ /home/wzj/app/kafka1/bin/kafka-run-class.sh kafka.tools.DumpLogSegments \
> --files 00000000000000000000.log \
> --print-data-log \
> > 0.txt
[wzj@hadoop001 bigdata-1]$ cat 0.txt
Dumping 00000000000000000000.log
Starting offset: 0

| offset: 0 CreateTime: 1590331130473 keysize: -1 valuesize: 1 sequence: -1 headerKeys: [] payload: 2
baseOffset: 1 lastOffset: 1 count: 1 baseSequence: -1 lastSequence: -1 producerId: -1 producerEpoch: -1 partitionLeaderEpoch: 0 isTransactional: false isControl: false position: 69 CreateTime: 1590331133059 size: 69 magic: 2 compresscodec: NONE crc: 3755360383 isvalid: true

| offset: 1 CreateTime: 1590331133059 keysize: -1 valuesize: 1 sequence: -1 headerKeys: [] payload: 5
baseOffset: 2 lastOffset: 2 count: 1 baseSequence: -1 lastSequence: -1 producerId: -1 producerEpoch: -1 partitionLeaderEpoch: 0 isTransactional: false isControl: false position: 138 CreateTime: 1590331135395 size: 69 magic: 2 compresscodec: NONE crc: 3920739238 isvalid: true

| offset: 2 CreateTime: 1590331135395 keysize: -1 valuesize: 1 sequence: -1 headerKeys: [] payload: 8
baseOffset: 3 lastOffset: 18 count: 16 baseSequence: -1 lastSequence: -1 producerId: -1 producerEpoch: -1 partitionLeaderEpoch: 1 isTransactional: false isControl: false position: 207 CreateTime: 1590850451836 size: 298 magic: 2 compresscodec: NONE crc: 3749326450 isvalid: true

| offset: 3 CreateTime: 1590850451834 keysize: -1 valuesize: 7 sequence: -1 headerKeys: [] payload: value_2
| offset: 4 CreateTime: 1590850451835 keysize: -1 valuesize: 7 sequence: -1 headerKeys: [] payload: value_5
| offset: 5 CreateTime: 1590850451835 keysize: -1 valuesize: 7 sequence: -1 headerKeys: [] payload: value_8
| offset: 6 CreateTime: 1590850451835 keysize: -1 valuesize: 8 sequence: -1 headerKeys: [] payload: value_11
| offset: 7 CreateTime: 1590850451835 keysize: -1 valuesize: 8 sequence: -1 headerKeys: [] payload: value_14
| offset: 8 CreateTime: 1590850451835 keysize: -1 valuesize: 8 sequence: -1 headerKeys: [] payload: value_17
| offset: 9 CreateTime: 1590850451835 keysize: -1 valuesize: 8 sequence: -1 headerKeys: [] payload: value_20
| offset: 10 CreateTime: 1590850451835 keysize: -1 valuesize: 8 sequence: -1 headerKeys: [] payload: value_23

index文件是索引文件，维护该partition的message对应的offset，对应的物理地址物理偏移量字节byte是【稀疏】维护,不是每个message都被维护到index

索引文件为什么不是全量的，而是稀疏的：

稀疏的索引文件有利用减少索引文件的大小，索引文件需要全部加载到内存中提高效率，根据稀疏程度可以减少几倍索引文件大小，然后进行少量的遍历即可找到数据。并且遍历数据时，是一个pagecache读取到内存中，此时极有可能会将后面需要的数据一块读入内存中

/home/wzj/app/kafka1/bin/kafka-run-class.sh kafka.tools.DumpLogSegments \
--files 00000000000000000000.index \
> 0.index

/home/wzj/app/kafka1/bin/kafka-run-class.sh kafka.tools.DumpLogSegments --files 00000000000000000000.index

欢迎关注公众号，一起愉快的交流
在这里插入图片描述