kafka的简单shell命令管理

启动kafka步骤
1、先启动zookeeper(kafka自带zookeeper的,可以启动自身的)当前目录为bin目录上一级
./bin/zookeeper-server-start.sh config/zookeeper.properties &
(启动后,基本上当前节点被日志占据了,后续操作另开窗口)

2、启动kafka服务   当前目录为bin目录上一级
./bin/kafka-server-start.sh config/server.properties &
(启动后,基本上当前节点被日志占据了,后续操作另开窗口)

3、创建一个1备份1partitions的topic名为steve 
./bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic steve


[root@VM_0_7_centos kafka2.10]# ./bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic stevetao
Error while executing topic command Topic "stevetao" already exists.
kafka.common.TopicExistsException: Topic "stevetao" already exists.
	at kafka.admin.AdminUtils$.createOrUpdateTopicPartitionAssignmentPathInZK(AdminUtils.scala:187)
	at kafka.admin.AdminUtils$.createTopic(AdminUtils.scala:172)
	at kafka.admin.TopicCommand$.createTopic(TopicCommand.scala:93)
	at kafka.admin.TopicCommand$.main(TopicCommand.scala:55)
	at kafka.admin.TopicCommand.main(TopicCommand.scala)

[root@VM_0_7_centos kafka2.10]# ./bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic steven
Created topic "steven".


查看所有topic列表

[root@hdp1 kafka-0.10.0]# ./bin/kafka-topics.sh --zookeeper 172.17.70.14:2181 --list


[root@VM_0_7_centos kafka2.10]# ./bin/kafka-topics.sh --zookeeper localhost:2181 --list
hive-mdatabase-hostsltable
steve
steven
stevetao


查看指定topic信息
[root@hdp1 kafka-0.10.0]# ./bin/kafka-topics.sh --zookeeper 172.17.70.14:2181 --topic IOT_DS_DATA(主题名) --describe


查看consumer group列表,使用--list参数
查看新版(信息保存在broker中)
 ./bin/kafka-consumer-groups.sh --new-consumer --bootstrap-server  172.17.70.14:6667(kafkaIP地址) --list

老版(信息保存在zookeeper中)
./bin/kafka-consumer-groups.sh --zookeeper  127.0.0.1:2181(zookeeperIP地址) --list

./bin/kafka-consumer-groups.sh --new-consumer --bootstrap-server  172.17.70.14:6667(kafkaIP地址) --group MongoDS(组名) --describe


控制台向topic生产数据   (steve为topic名;)         Alt+C可以结束操作

./bin/kafka-console-producer.sh --broker-list localhost:9092 --topic steve

[root@VM_0_7_centos kafka2.10]# ./bin/kafka-console-producer.sh --broker-list localhost:9092 --topic steve
[2018-04-16 15:47:54,875] WARN Property topic is not valid (kafka.utils.VerifiableProperties)
dsfdfd
zhangsan
lisi


控制台消费topic的数据(steve为topic名;)
./bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic steve --from-beginning


[root@VM_0_7_centos kafka2.10]# ./bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic steve --from-beginning
zdfd
dfadfaf\
dfadf
stevev
stevetao
zz
zhngsan
dsfdfd
zhangsan
lisi		^C Consumed 10 messages


查看topic某分区偏移量最大(小)值(steve为topic名;分区从0开始;time为-1时表示最大值,time为-2时表示最小值)
./bin/kafka-run-class.sh kafka.tools.GetOffsetShell --topic steve --time -1 --broker-list localhost:9092 --partitions  0

[root@VM_0_7_centos kafka2.10]# ./bin/kafka-run-class.sh kafka.tools.GetOffsetShell --topic steve --time -1 --broker-list localhost:9092 --partitions 0
steve:0:7


增加topic分区数  10
bin/kafka-topics.sh --zookeeper localhost:2181  --alter --topic steve --partitions 10


删除topic,慎用,只会删除zookeeper中的元数据,消息文件须手动删除
bin/kafka-run-class.sh kafka.admin.DeleteTopicCommand --zookeeper localhost:2181 --topic steve


查看topic消费进度
bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker
阅读更多
想对作者说点什么? 我来说一句

没有更多推荐了,返回首页