kafka基本使用

借助《Kafka单机部署》和《Kafka集群部署》创建的集群环境

查询所有主题

# pwd
/usr/local/kafka/bin
# ./kafka-run-class.sh kafka.admin.TopicCommand --zookeeper test_1.thinking.com:2181 --list

创建主题

有两种方式创建主题。

第一种是隐式的

如果broker的server. properties文件里面配置的auto.create.topics.enable=true时（这个值默认是true），当生产者向一个尚未创建的主题发送消息时，会自动创建一个拥有${num.partitions}个分区和${default.replication.factor}个副本的主题。

例如在我们的机器中

auto.create.topics.enable没找到

num.partitions=1

default.replication.factor没找到

我们在单机环境上试一下

# ./kafka-console-producer.sh --broker-list kafka.thinking.com:9092 --topic test-no-create
>Test Msg 1001
>Test Msg 1002
# ./kafka-run-class.sh kafka.admin.TopicCommand --zookeeper test_1.thinking.com:2181 --list
test-no-create
# ./kafka-topics.sh --zookeeper test_1.thinking.com:2181 --describe --topic test-no-create
Topic:test-no-create	PartitionCount:1	ReplicationFactor:1	Configs:
	Topic: test-no-create	Partition: 0	Leader: 1	Replicas: 1	Isr: 1

执行结果显示，直接向一个尚未创建的主题发送消息时，确实会自动创建这个主题。

第二种方式是显式的

在单机环境中，创建命令如：

# ./kafka-topics.sh --create --zookeeper test_1.thinking.com:2181 --replication-factor 1 --partitions 3 --topic test-with-create
Created topic "test-with-create".
# ./kafka-run-class.sh kafka.admin.TopicCommand --zookeeper test_1.thinking.com:2181 --list
test-no-create
test-with-create
# ./kafka-topics.sh --zookeeper test_1.thinking.com:2181 --describe --topic test-with-create
Topic:test-with-create	PartitionCount:3	ReplicationFactor:1	Configs:
	Topic: test-with-create	Partition: 0	Leader: 1	Replicas: 1	Isr: 1
	Topic: test-with-create	Partition: 1	Leader: 1	Replicas: 1	Isr: 1
	Topic: test-with-create	Partition: 2	Leader: 1	Replicas: 1	Isr: 1

这里我们创建了一个主题test-with-create。这个主题有3个分区（由--partitions 3指定），有1个副本（由--replication-factor 1指定）。

通过查询我们得知，test-with-create的三个分区（test-with-create    Partition: 0/1/2）全部位于broker.id=1的服务器上（Leader: 1）

在broker的server. properties文件里面可以找到

log.dirs=/tmp/kafka-logs

进到${log.dirs}目录下，可以看到

# pwd
/tmp/kafka-logs
# ll
total 52
drwxr-xr-x 9 root root 4096 Aug  8 06:46 ./
drwxrwxrwt 1 root root 4096 Aug  8 05:35 ../
-rw-r--r-- 1 root root    0 Aug  8 05:35 .lock
-rw-r--r-- 1 root root    0 Aug  8 05:35 cleaner-offset-checkpoint
-rw-r--r-- 1 root root    4 Aug  8 06:46 log-start-offset-checkpoint
-rw-r--r-- 1 root root   54 Aug  8 05:35 meta.properties
-rw-r--r-- 1 root root  149 Aug  8 06:46 recovery-point-offset-checkpoint
-rw-r--r-- 1 root root  149 Aug  8 06:46 replication-offset-checkpoint
drwxr-xr-x 2 root root 4096 Aug  8 06:29 test-no-create-0/
drwxr-xr-x 2 root root 4096 Aug  8 06:38 test-with-create-0/
drwxr-xr-x 2 root root 4096 Aug  8 06:38 test-with-create-1/
drwxr-xr-x 2 root root 4096 Aug  8 06:38 test-with-create-2/

我们在集群环境中尝试一下同样的命令，我们进入集群的任意一个服务器

$ docker exec -it kafka_2.thinking.com /bin/bash
# ./kafka-topics.sh --create --zookeeper test_1.thinking.com:2181 --replication-factor 1 --partitions 3 --topic test-with-create
Created topic "test-with-create".
# ./kafka-topics.sh --zookeeper test_1.thinking.com:2181 --describe --topic test-with-create
Topic:test-with-create	PartitionCount:3	ReplicationFactor:1	Configs:
	Topic: test-with-create	Partition: 0	Leader: 2	Replicas: 2	Isr: 2
	Topic: test-with-create	Partition: 1	Leader: 3	Replicas: 3	Isr: 3
	Topic: test-with-create	Partition: 2	Leader: 1	Replicas: 1	Isr: 1

从这里可以看到，test-with-create的三个分区均匀地分布于三台服务器上。

分区0位于broker.id=2的机器上；

分区1位于broker.id=3的机器上；

分区2位于broker.id=1的机器上；

在broker.id=2的机器${log.dirs}目录下，可以看到

# ll /tmp/kafka-logs
total 28
drwxr-xr-x 3 root root 4096 Aug  8 08:34 ./
drwxrwxrwt 1 root root 4096 Aug  8 07:56 ../
-rw-r--r-- 1 root root    0 Aug  8 07:56 .lock
-rw-r--r-- 1 root root    0 Aug  8 07:56 cleaner-offset-checkpoint
-rw-r--r-- 1 root root    4 Aug  8 08:33 log-start-offset-checkpoint
-rw-r--r-- 1 root root   54 Aug  8 07:56 meta.properties
-rw-r--r-- 1 root root   25 Aug  8 08:33 recovery-point-offset-checkpoint
-rw-r--r-- 1 root root   25 Aug  8 08:34 replication-offset-checkpoint
drwxr-xr-x 2 root root 4096 Aug  8 08:27 test-with-create-0/

进一步地，我们将副本数设置成2，看看分区的分布情况

# ./kafka-topics.sh --create --zookeeper test_1.thinking.com:2181 --replication-factor 2 --partitions 3 --topic test-with-create-2
Created topic "test-with-create-2".
# ./kafka-topics.sh --zookeeper test_1.thinking.com:2181 --describe --topic test-with-create-2
	Topic: test-with-create-2	Partition: 0	Leader: 3	Replicas: 3,1	Isr: 3,1
	Topic: test-with-create-2	Partition: 1	Leader: 1	Replicas: 1,2	Isr: 1,2
	Topic: test-with-create-2	Partition: 2	Leader: 2	Replicas: 2,3	Isr: 2,3

可以看到：

分区0 位于broker.id=3和broker.id=1的机器上，其中broker.id=3的机器是首领

分区1 位于broker.id=1和broker.id=2的机器上，其中broker.id=1的机器是首领

分区2 位于broker.id=2和broker.id=3的机器上，其中broker.id=2的机器是首领