Kafka--下载与安装

下载地址

Apache Kafka 官网：https://kafka.apache.org/downloads.html
下载：
Scala 2.13 - kafka_2.13-2.7.0.tgz (asc, sha512)

1. 解压

> tar -xzf kafka_2.13-2.7.0.tgz
> cd kafka_2.13-2.7.0

2.启动服务

启动Kafka自带打包和配置好的Zookeeper。

> bin/zookeeper-server-start.sh config/zookeeper.properties
[2021-02-28 15:48:57,147] INFO Reading configuration from: config/zookeeper.properties (org.apache.zookeeper.server.quorum.QuorumPeerConfig)

启动Kafka服务

> bin/kafka-server-start.sh config/server.properties &
...

3. 创建一个topic

创建一个名为“test”的topic，1个分区，1个副本

bin/kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --topic test

创建成功

Created topic test.

查看topic列表

> bin/kafka-topics.sh --list --zookeeper localhost:2181
test

4. 发送消息

打开一个终端窗口，执行命令，再输入内容发送消息到test中

> bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test
This is a message
This is another message

5.消费消息

另打开一个终端，执行命令，就可以消费到topic中的消息并再控制台输出

bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning
This is a message
This is another message

如果你有2台不同的终端上运行上述命令，那么当你在运行生产者时，消费者就能消费到生产者发送的消息。

6.设置多个broker

step1:复制两个配置文件

> cp config/server.properties config/server-1.properties 
> cp config/server.properties config/server-2.properties

修改配置文件以下内容：

config/server-1.properties: 
    broker.id=1 
    listeners=PLAINTEXT://:9093 
    log.dir=/tmp/kafka-logs-1

config/server-2.properties: 
    broker.id=2 
    listeners=PLAINTEXT://:9094 
    log.dir=/tmp/kafka-logs-2

broker.id是集群中每个节点的唯一且永久的名称，我们修改端口和日志目录是因为我们现在在同一台机器上运行，我们要防止broker在同一端口上注册和覆盖对方的数据。
启动刚配置的两个节点：

> bin/kafka-server-start.sh config/server-1.properties &
... 
> bin/kafka-server-start.sh config/server-2.properties &
...

创建一个新的topic，3个副本：

> bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 3 --partitions 1 --topic my-replicated-topic

查看集群的信息：

> bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic my-replicated-topic
Topic: my-replicated-topic	PartitionCount: 1	ReplicationFactor: 3	Configs: 
Topic: my-replicated-topic	Partition: 0	Leader: 2	Replicas: 2,0,1	Isr: 2,0,1

输出解释：第一行是所有分区的摘要，其次，每一行提供一个分区信息，因为我们只有一个分区，所以只有一行。

• “leader”：该节点负责该分区的所有的读和写，每个节点的leader都是随机选择的。
• “replicas”：备份的节点列表，无论该节点是否是leader或者目前是否还活着，只是显示。
• “isr”：“同步备份”的节点列表，也就是活着的节点并且正在同步leader。

发送消息：

bin/kafka-console-producer.sh --broker-list localhost:9092 --topic my-replicated-topic
>hello world
>hello World!

消费消息：

bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --from-beginning --topic my-replicated-topic
hello world
hello World!

检测集群的容错能力，kill主节点：

> ps | grep server-2.properties
10011 ttys001    0:20.15 /Library/Java/JavaVirtualMachines/jdk1.8.0_161.j...
> kill -9 10011

此时可以看到控制台有错误日志：

[ReplicaFetcher replicaId=1, leaderId=2, fetcherId=0] Error sending fetch request (sessionId=1828535184, epoch=1482) to node 2: (org.apache.kafka.clients.FetchSessionHandler)
java.io.IOException: Connection to 2 was disconnected before the response was read

但是过一会报错就会停止：

[2021-02-28 16:38:40,890] INFO [ReplicaFetcherManager on broker 0] Removed fetcher for partitions Set(my-replicated-topic-0) (kafka.server.ReplicaFetcherManager)
[2021-02-28 16:38:40,891] INFO [ReplicaFetcherManager on broker 1] Removed fetcher for partitions HashSet(my-replicated-topic-0) (kafka.server.ReplicaFetcherManager)
[2021-02-28 16:38:40,893] INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Shutting down (kafka.server.ReplicaFetcherThread)
[2021-02-28 16:38:40,894] INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Stopped (kafka.server.ReplicaFetcherThread)
[2021-02-28 16:38:40,894] INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Shutdown completed (kafka.server.ReplicaFetcherThread)
[2021-02-28 16:38:40,895] INFO [ReplicaFetcher replicaId=1, leaderId=0, fetcherId=0] Starting (kafka.server.ReplicaFetcherThread)
[2021-02-28 16:38:40,895] INFO [ReplicaFetcherManager on broker 1] Added fetcher to broker 0 for partitions Map(my-replicated-topic-0 -> (offset=2, leaderEpoch=1)) (kafka.server.ReplicaFetcherManager)
[2021-02-28 16:38:40,896] INFO [ReplicaFetcher replicaId=1, leaderId=2, fetcherId=0] Shutting down (kafka.server.ReplicaFetcherThread)
[2021-02-28 16:38:40,897] INFO [ReplicaFetcher replicaId=1, leaderId=2, fetcherId=0] Shutdown completed (kafka.server.ReplicaFetcherThread)
[2021-02-28 16:38:40,898] INFO [ReplicaFetcher replicaId=1, leaderId=2, fetcherId=0] Stopped (kafka.server.ReplicaFetcherThread)
[2021-02-28 16:38:40,909] INFO [Log partition=my-replicated-topic-0, dir=/tmp/kafka-logs-1] Truncating to 2 has no effect as the largest offset in the log is 1 (kafka.log.Log)

查看集群现在的情况：

> bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic my-replicated-topic
Topic: my-replicated-topic	PartitionCount: 1	ReplicationFactor: 3	Configs: 
Topic: my-replicated-topic	Partition: 0	Leader: 0	Replicas: 2,0,1	Isr: 0,1

消费消息，发现消息没有丢失：

bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --from-beginning --topic my-replicated-topic
hello world
hello World!