Kafka MQ 安装以及使用指南

Kafka 是由 Apache 软件基金会开发的一个开源流处理平台，由 Scala 和 Java 编写。该项目的目标是为处理实时数据提供一个统一、高吞吐、低延迟的平台。

下载

https://downloads.apache.org/kafka/3.6.1/kafka_2.13-3.6.1.tgz

安装

tar xf kafka_2.13-3.6.1.tgz
cd kafka_2.13-3.6.1

启动

本地必须具备 JAVA 8+ 运行环境

# 启动 ZooKeeper 应用
bin/zookeeper-server-start.sh config/zookeeper.properties

# 启动 Kafka 应用
bin/kafka-server-start.sh config/server.properties

创建主题(TOPIC)

# 主题名称为:test-events
bin/kafka-topics.sh --create --topic test-events --bootstrap-server localhost:9092
​
# 列出当前有哪些主题
bin/kafka-topics.sh --bootstrap-server=localhost:9092 --list

向主题写入事件

bin/kafka-console-producer.sh --topic test-events --bootstrap-server localhost:9092
This is my first event
This is my second event
​
# Ctrl-C 停止写入

读取事件

bin/kafka-console-consumer.sh --topic test-events --from-beginning --bootstrap-server localhost:9092
This is my first event
This is my second event
​
# Ctrl-C 停止读取

Systemd 方式运行 ZooKeeper 和 Kafka

1. vim /etc/systemd/system/kafka-zookeeper.service

   [Unit]
   Description=Apache Zookeeper server (Kafka)
   Documentation=http://zookeeper.apache.org
   Requires=network.target remote-fs.target
   After=network.target remote-fs.target
   ​
   [Service]
   Type=simple
   User=root
   Group=root
   Environment=JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk
   ExecStart=/opt/kafka/bin/zookeeper-server-start.sh /opt/kafka/config/zookeeper.properties
   ExecStop=/opt/kafka/bin/zookeeper-server-stop.sh
   ​
   [Install]
   WantedBy=multi-user.target

2. vim /etc/systemd/system/kafka.service

   [Unit]
   Description=Apache Kafka server (broker)
   Documentation=http://kafka.apache.org/documentation.html
   Requires=network.target remote-fs.target
   After=network.target remote-fs.target kafka-zookeeper.service
   ​
   [Service]
   Type=simple
   User=root
   Group=root
   Environment=JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk
   ExecStart=/opt/kafka/bin/kafka-server-start.sh /opt/kafka/config/server.properties
   ExecStop=/opt/kafka/bin/kafka-server-stop.sh
   ​
   [Install]
   WantedBy=multi-user.target

3. vim /opt/kafka/config/server.properties

   listeners=PLAINTEXT://0.0.0.0:9092

4. 重新启动服务

   systemctl daemon-reload
   systemctl start kafka-zookeeper.service
   systemctl start kafka.service