上篇文章[Kafka]安装与部署中，搭建了一个三节点的Kafka集群环境。
这篇文章来谈谈如何使用命令行操作集群。
topic创建
创建一个名为test01的topic
bin/kafka-topics.sh \
--create \
--zookeeper 192.168.133.13:2181,192.168.133.14:2181,192.168.133.15:2181 \
--replication-factor 2 \
--partitions 3 \
--topic test01
--zookeeper zookeeper的连接地址
--replication-factor num 指定数据副本的数量为num
--partitions num 指定该topic有几个主分片
--topic name 指定topic的名称
topic查询
查看所有topic
[app@test13 kafka]$ bin/kafka-topics.sh --list --zookeeper 192.168.133.13:2181,192.168.133.14:2181,192.168.133.15:2181
test
test01
test02
该命令可以查看所有zookeeper中的topic。
查看指定topic明细
bin/kafka-topics.sh \
--describe \
--zookeeper 192.168.133.13:2181,192.168.133.14:2181,192.168.133.15:2181 \
--topic test01
查看test01的topic的详细信息：
[app@test13 kafka]$ bin/kafka-topics.sh \
> --describe \
> --zookeeper 192.168.133.13:2181,192.168.133.14:2181,192.168.133.15:2181 \
> --topic test01
Topic:test01 PartitionCount:3 ReplicationFactor:2 Configs:
Topic: test01 Partition: 0 Leader: 1 Replicas: 1,2 Isr: 1,2
Topic: test01 Partition: 1 Leader: 0 Replicas: 2,0 Isr: 0,2
Topic: test01 Partition: 2 Leader: 0 Replicas: 0,1 Isr: 0,1
简单解释一下输出，topic是消息队列名，PartitionCount是分片数量其实也就是数据分布在几个broker上面，ReplicationFactor是副本数量。
具体的信息输出显示了有哪些分片，存在于哪个broker上面，该分片上还有哪些副本，ISR这个集合中的所有节点都是存活状态，并且跟leader同步。
topic修改
可进行修改分区数量的操作，也就是指定partitions。
先看一下原来的topic信息：
[app@test13 kafka]$ bin/kafka-topics.sh \
> --describe \
> --zookeeper 192.168.133.13:2181,192.168.133.14:2181,192.168.133.15:2181 \
> --topic test
Topic:test PartitionCount:3 ReplicationFactor:3 Configs:
Topic: test Partition: 0 Leader: 0 Replicas: 0,1,2 Isr: 0,1,2
Topic: test Partition: 1 Leader: 1 Replicas: 1,2,0 Isr: 1,2,0
Topic: test Partition: 2 Leader: 2 Replicas: 2,0,1 Isr: 2,0,1
现在进行修改，将分片数改为2：
[app@test13 kafka]$ bin/kafka-topics.sh \
> --alter \
> --zookeeper 192.168.133.13:2181,192.168.133.14:2181,192.168.133.15:2181 \
> --topic test --partitions 2
WARNING: If partitions are increased for a topic that has a key, the partition logic or ordering of the messages will be affected
Error while executing topic command : The number of partitions for a topic can only be increased. Topic test currently has 3 partitions, 2 would not be an increase.
[2019-11-12 16:11:32,686] ERROR org.apache.kafka.common.errors.InvalidPartitionsException: The number of partitions for a topic can only be increased. Topic test currently has 3 partitions, 2 would not be an increase.
(kafka.admin.TopicCommand$)
从上面执行结果看，失败了，原因是分区数只能增加不能减少。这种场景一般适用于加了几台broker，然后进行分区扩容。
这里修改一下partition数量，修改为4：
[app@test13 kafka]$ bin/kafka-topics.sh --alter --zookeeper 192.168.133.13:2181,192.168.133.14:2181,192.168.133.15:2181 --topic test --partitions 4
WARNING: If partitions are increased for a topic that has a key, the partition logic or ordering of the messages will be affected
Adding partitions succeeded!
再查看具体信息：
[app@test13 kafka]$ bin/kafka-topics.sh --describe --zookeeper 192.168.133.13:2181,192.168.133.14:2181,192.168.133.15:2181 --topic test
Topic:test PartitionCount:4 ReplicationFactor:3 Configs:
Topic: test Partition: 0 Leader: 0 Replicas: 0,1,2 Isr: 0,1,2
Topic: test Partition: 1 Leader: 1 Replicas: 1,2,0 Isr: 1,2,0
Topic: test Partition: 2 Leader: 2 Replicas: 2,0,1 Isr: 2,0,1
Topic: test Partition: 3 Leader: 0 Replicas: 0,2,1 Isr: 0,2,1
topic删除
原本的集群中有两个topic——test01和test02：
[app@test13 kafka]$ bin/kafka-topics.sh --list --zookeeper 192.168.133.13:2181,192.168.133.14:2181,192.168.133.15:2181
test01
test02
现在删除名为test01的topic：
[app@test13 kafka]$ bin/kafka-topics.sh --delete --zookeeper 192.168.133.13:2181,192.168.133.14:2181,192.168.133.15:2181 --topic test01
Topic test01 is marked for deletion.
Note: This will have no impact if delete.topic.enable is not set to true.
[app@test13 kafka]$ bin/kafka-topics.sh --list --zookeeper 192.168.133.13:2181,192.168.133.14:2181,192.168.133.15:2181
test02
[app@test13 kafka]$
这里删除是先标记为删除，由kafka内部 scheduled for deletion。
生产者
kafka-console-producer.sh命令使用：
This tool helps to read data from standard input and publish it to Kafka.
Option Description
------ -----------
--batch-size <Integer: size> Number of messages to send in a single
batch if they are not being sent
synchronously. (default: 200)
--broker-list <String: broker-list> REQUIRED: The broker list string in
the form HOST1:PORT1,HOST2:PORT2.
--compression-codec [String: The compression codec: either 'none',
compression-codec] 'gzip', 'snappy', 'lz4', or 'zstd'.
If specified without value, then it
defaults to 'gzip'
--help Print usage information.
--line-reader <String: reader_class> The class name of the class to use for
reading lines from standard in. By
default each line is read as a
separate message. (default: kafka.
tools.
ConsoleProducer$LineMessageReader)
--max-block-ms <Long: max block on The max time that the producer will
send> block for during a send request
(default: 60000)
--max-memory-bytes <Long: total memory The total memory used by the producer
in bytes> to buffer records waiting to be sent
to the server. (default: 33554432)
--max-partition-memory-bytes <Long: The buffer size allocated for a
memory in bytes per partition> partition. When records are received
which are smaller than this size the
producer will attempt to
optimistically group them together
until this size is reached.
(default: 16384)
--message-send-max-retries <Integer> Brokers can fail receiving the message
for multiple reasons, and being
unavailable transiently is just one
of them. This property specifies the
number of retires before the
producer give up and drop this
message. (default: 3)
--metadata-expiry-ms <Long: metadata The period of time in milliseconds
expiration interval> after which we force a refresh of
metadata even if we haven't seen any
leadership changes. (default: 300000)
--producer-property <String: A mechanism to pass user-defined
producer_prop> properties in the form key=value to
the producer.
--producer.config <String: config file> Producer config properties file. Note
that [producer-property] takes
precedence over this config.
--property <String: prop> A mechanism to pass user-defined
properties in the form key=value to
the message reader. This allows
custom configuration for a user-
defined message reader.
--request-required-acks <String: The required acks of the producer
request required acks> requests (default: 1)
--request-timeout-ms <Integer: request The ack timeout of the producer
timeout ms> requests. Value must be non-negative
and non-zero (default: 1500)
--retry-backoff-ms <Integer> Before each retry, the producer
refreshes the metadata of relevant
topics. Since leader election takes
a bit of time, this property
specifies the amount of time that
the producer waits before refreshing
the metadata. (default: 100)
--socket-buffer-size <Integer: size> The size of the tcp RECV size.
(default: 102400)
--sync If set message send requests to the
brokers are synchronously, one at a
time as they arrive.
--timeout <Integer: timeout_ms> If set and the producer is running in
asynchronous mode, this gives the
maximum amount of time a message
will queue awaiting sufficient batch
size. The value is given in ms.
(default: 1000)
--topic <String: topic> REQUIRED: The topic id to produce
messages to.
--version Display Kafka version.
先创建一个topic名为test01的消息队列，然后开启生产者：
bin/kafka-console-producer.sh --broker-list 192.168.133.15:9092 --topic test01
消费者
[app@node14 kafka]$ bin/kafka-console-consumer.sh --help
This tool helps to read data from Kafka topics and outputs it to standard output.
Option Description
------ -----------
--bootstrap-server <String: server to REQUIRED: The server(s) to connect to.
connect to>
--consumer-property <String: A mechanism to pass user-defined
consumer_prop> properties in the form key=value to
the consumer.
--consumer.config <String: config file> Consumer config properties file. Note
that [consumer-property] takes
precedence over this config.
--enable-systest-events Log lifecycle events of the consumer
in addition to logging consumed
messages. (This is specific for
system tests.)
--formatter <String: class> The name of a class to use for
formatting kafka messages for
display. (default: kafka.tools.
DefaultMessageFormatter)
--from-beginning If the consumer does not already have
an established offset to consume
from, start with the earliest
message present in the log rather
than the latest message.
--group <String: consumer group id> The consumer group id of the consumer.
--help Print usage information.
--isolation-level <String> Set to read_committed in order to
filter out transactional messages
which are not committed. Set to
read_uncommittedto read all
messages. (default: read_uncommitted)
--key-deserializer <String:
deserializer for key>
--max-messages <Integer: num_messages> The maximum number of messages to
consume before exiting. If not set,
consumption is continual.
--offset <String: consume offset> The offset id to consume from (a non-
negative number), or 'earliest'
which means from beginning, or
'latest' which means from end
(default: latest)
--partition <Integer: partition> The partition to consume from.
Consumption starts from the end of
the partition unless '--offset' is
specified.
--property <String: prop> The properties to initialize the
message formatter. Default
properties include:
print.timestamp=true|false
print.key=true|false
print.value=true|false
key.separator=<key.separator>
line.separator=<line.separator>
key.deserializer=<key.deserializer>
value.deserializer=<value.
deserializer>
Users can also pass in customized
properties for their formatter; more
specifically, users can pass in
properties keyed with 'key.
deserializer.' and 'value.
deserializer.' prefixes to configure
their deserializers.
--skip-message-on-error If there is an error when processing a
message, skip it instead of halt.
--timeout-ms <Integer: timeout_ms> If specified, exit if no message is
available for consumption for the
specified interval.
--topic <String: topic> The topic id to consume on.
--value-deserializer <String:
deserializer for values>
--version Display Kafka version.
--whitelist <String: whitelist> Regular expression specifying
whitelist of topics to include for
consumption.
这里开启两个消费者（192.168.133.13,192.168.133.14），在一个消费者组内：
[app@node14 kafka]$ bin/kafka-console-consumer.sh --bootstrap-server 192.168.133.13:9092 --group hello --topic test01
1.发送消息
2.接收消息
在192.168.133.13上面接收到以下消息：
在192.168.133.14上面接收到以下消息：
4680
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
