下载地址
Apache Kafka 官网:https://kafka.apache.org/downloads.html
下载:
Scala 2.13 - kafka_2.13-2.7.0.tgz (asc, sha512)
1. 解压
> tar -xzf kafka_2.13-2.7.0.tgz
> cd kafka_2.13-2.7.0
2.启动服务
启动Kafka自带打包和配置好的Zookeeper。
> bin/zookeeper-server-start.sh config/zookeeper.properties
[2021-02-28 15:48:57,147] INFO Reading configuration from: config/zookeeper.properties (org.apache.zookeeper.server.quorum.QuorumPeerConfig)
启动Kafka服务
> bin/kafka-server-start.sh config/server.properties &
...
3. 创建一个topic
创建一个名为“test”的topic,1个分区,1个副本
bin/kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --topic test
创建成功
Created topic test.
查看topic列表
> bin/kafka-topics.sh --list --zookeeper localhost:2181
test
4. 发送消息
打开一个终端窗口,执行命令,再输入内容发送消息到test中
> bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test
This is a message
This is another message
5.消费消息
另打开一个终端,执行命令,就可以消费到topic中的消息并再控制台输出
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning
This is a message
This is another message
如果你有2台不同的终端上运行上述命令,那么当你在运行生产者时,消费者就能消费到生产者发送的消息。
6.设置多个broker
step1:复制两个配置文件
> cp config/server.properties config/server-1.properties
> cp config/server.properties config/server-2.properties
修改配置文件以下内容:
config/server-1.properties:
broker.id=1
listeners=PLAINTEXT://:9093
log.dir=/tmp/kafka-logs-1
config/server-2.properties:
broker.id=2
listeners=PLAINTEXT://:9094
log.dir=/tmp/kafka-logs-2
broker.id是集群中每个节点的唯一且永久的名称,我们修改端口和日志目录是因为我们现在在同一台机器上运行,我们要防止broker在同一端口上注册和覆盖对方的数据。
启动刚配置的两个节点:
> bin/kafka-server-start.sh config/server-1.properties &
...
> bin/kafka-server-start.sh config/server-2.properties &
...
创建一个新的topic,3个副本:
> bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 3 --partitions 1 --topic my-replicated-topic
查看集群的信息:
> bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic my-replicated-topic
Topic: my-replicated-topic PartitionCount: 1 ReplicationFactor: 3 Configs:
Topic: my-replicated-topic Partition: 0 Leader: 2 Replicas: 2,0,1 Isr: 2,0,1
输出解释:第一行是所有分区的摘要,其次,每一行提供一个分区信息,因为我们只有一个分区,所以只有一行。
- “leader”:该节点负责该分区的所有的读和写,每个节点的leader都是随机选择的。
- “replicas”:备份的节点列表,无论该节点是否是leader或者目前是否还活着,只是显示。
- “isr”:“同步备份”的节点列表,也就是活着的节点并且正在同步leader。
发送消息:
bin/kafka-console-producer.sh --broker-list localhost:9092 --topic my-replicated-topic
>hello world
>hello World!
消费消息:
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --from-beginning --topic my-replicated-topic
hello world
hello World!
检测集群的容错能力,kill主节点:
> ps | grep server-2.properties
10011 ttys001 0:20.15 /Library/Java/JavaVirtualMachines/jdk1.8.0_161.j...
> kill -9 10011
此时可以看到控制台有错误日志:
[ReplicaFetcher replicaId=1, leaderId=2, fetcherId=0] Error sending fetch request (sessionId=1828535184, epoch=1482) to node 2: (org.apache.kafka.clients.FetchSessionHandler)
java.io.IOException: Connection to 2 was disconnected before the response was read
但是过一会报错就会停止:
[2021-02-28 16:38:40,890] INFO [ReplicaFetcherManager on broker 0] Removed fetcher for partitions Set(my-replicated-topic-0) (kafka.server.ReplicaFetcherManager)
[2021-02-28 16:38:40,891] INFO [ReplicaFetcherManager on broker 1] Removed fetcher for partitions HashSet(my-replicated-topic-0) (kafka.server.ReplicaFetcherManager)
[2021-02-28 16:38:40,893] INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Shutting down (kafka.server.ReplicaFetcherThread)
[2021-02-28 16:38:40,894] INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Stopped (kafka.server.ReplicaFetcherThread)
[2021-02-28 16:38:40,894] INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Shutdown completed (kafka.server.ReplicaFetcherThread)
[2021-02-28 16:38:40,895] INFO [ReplicaFetcher replicaId=1, leaderId=0, fetcherId=0] Starting (kafka.server.ReplicaFetcherThread)
[2021-02-28 16:38:40,895] INFO [ReplicaFetcherManager on broker 1] Added fetcher to broker 0 for partitions Map(my-replicated-topic-0 -> (offset=2, leaderEpoch=1)) (kafka.server.ReplicaFetcherManager)
[2021-02-28 16:38:40,896] INFO [ReplicaFetcher replicaId=1, leaderId=2, fetcherId=0] Shutting down (kafka.server.ReplicaFetcherThread)
[2021-02-28 16:38:40,897] INFO [ReplicaFetcher replicaId=1, leaderId=2, fetcherId=0] Shutdown completed (kafka.server.ReplicaFetcherThread)
[2021-02-28 16:38:40,898] INFO [ReplicaFetcher replicaId=1, leaderId=2, fetcherId=0] Stopped (kafka.server.ReplicaFetcherThread)
[2021-02-28 16:38:40,909] INFO [Log partition=my-replicated-topic-0, dir=/tmp/kafka-logs-1] Truncating to 2 has no effect as the largest offset in the log is 1 (kafka.log.Log)
查看集群现在的情况:
> bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic my-replicated-topic
Topic: my-replicated-topic PartitionCount: 1 ReplicationFactor: 3 Configs:
Topic: my-replicated-topic Partition: 0 Leader: 0 Replicas: 2,0,1 Isr: 0,1
消费消息,发现消息没有丢失:
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --from-beginning --topic my-replicated-topic
hello world
hello World!