EMQX与kafka和MQTT结合

参考github原版

链接：https://github.com/SINTEF-9012/kafka-mqtt-source-connector

1、 下载EMQX

Mqtt Broker - EMQX

• Download EMQX from https://www.emqx.io/downloads
• Extract the download to your desired destination, here termed “path-to-emqx”.
• Run the following command in your terminal to start the EMQX broker:

"path-to-emqx"/emqx/bin/emqx start

/home/jiayuan/apps/emqx/bin/emqx start

• Check that EMQX is running with the following terminal command:

"path-to-emqx"/emqx/bin/emqx_ctl status

/home/jiayuan/apps/emqx/bin/emqx_ctl status

2. 下载kafka

Download Kafka binaries

Download a binary Kafka release from https://kafka.apache.org/downloads. We work with the compressed download:

kafka_2.13-2.4.1.tgz Extract the download to your desired destination, here termed “path-to-kafka”.

Zookeeper

About Zookeeper:

“Zookeeper is a top-level software developed by Apache that acts as a centralized service and is used to maintain naming and configuration data and to provide flexible and robust synchronization within distributed systems. Zookeeper keeps track of status of the Kafka cluster nodes and it also keeps track of Kafka topics, partitions etc. Zookeeper it self is allowing multiple clients to perform simultaneous reads and writes and acts as a shared configuration service within the system. The Zookeeper atomic broadcast (ZAB) protocol i s the brains of the whole system, making it possible for Zookeeper to act as an atomic broadcast system and issue orderly updates.” Cloudkarafka

Start Zookeeper

启动zookeeper

"path-to-kafka"/kafka_2.13-2.4.1/bin/zookeeper-server-start.sh "path-to-kafka"/kafka_2.13-2.4.1/config/zookeeper.properties

/home/jiayuan/apps/kafka_2.13-2.4.1

"path-to-kafka"/kafka_2.13-2.4.1/bin/zookeeper-server-start.sh "path-to-kafka"/kafka_2.13-2.4.1/config/zookeeper.properties

P.S. The default properties of zookeeper.properties works well for this tutorial’s purpose. It will start Zookeeper on the default port 2181.

Kafka Broker

As mentioned, we will only kick up a single instance Kafka Broker. The Kafka Broker will use "path-to-kafka"/kafka_2.13-2.4.1/config/server.properties, and it could be worth checking that

修改kafka连接zookeeper端口，默认即可

zookeeper.connect=localhost:2181

or set according to your custom configuration in zookeeper.properties.

Start Kafka Broker

启动kafka

"path-to-kafka"/kafka_2.13-2.4.1/bin/kafka-server-start.sh "path-to-kafka"/kafka_2.13-2.4.1/config/server.properties

/opt/software/kafka_2.13-2.4.1/bin/kafka-server-start.sh  /opt/software/kafka_2.13-2.4.1/config/server.properties

/home/jiayuan/apps/kafka_2.13-2.4.1/bin/kafka-server-start.sh     /home/jiayuan/apps/kafka_2.13-2.4.1/config/server.properties

Create Kafka Topic**

创建topic

"path-to-kafka"/kafka_2.13-2.4.1/bin/kafka-topics --create --bootstrap-server 127.0.0.1:9092 --replication-factor 1 --partitions 1 --topic test


/opt/software/kafka_2.13-2.4.1/bin/kafka-topics.sh --create --bootstrap-server 127.0.0.1:9092 --replication-factor 1 --partitions 1 --topic test



/home/jiayuan/apps/kafka_2.13-2.4.1/bin/kafka-topics.sh --create --bootstrap-server 127.0.0.1:9092 --replication-factor 1 --partitions 1 --topic test

查看某个Topic的详情

/opt/software/kafka_2.13-2.4.1/bin/kafka-topics.sh --zookeeper localhost:2181 --describe --topic test

/home/jiayuan/apps/kafka_2.13-2.4.1/bin/kafka-topics.sh --zookeeper localhost:2181 --describe --topic test

3. 克隆或者下载仓库

git clone https://github.com/SINTEF-9012/kafka-mqtt-source-connector.git

Kafka Connect

The Kafka Connect API is what we utilise as a framework around our connectors, to handle scaling, polling from Kafka, work distribution etc. Kafka Connect can run as connect-standalone or as connect-distributed. The connect-standalone is engineered for demo and test purposes, as it cannot provide fallback in a production environment.

Start Kafka Connect Follow the respective steps below to start Kafka Connect in preferred mode.

Connect in general Build this java maven project, but navigating to root kafka-mqtt-source-connector in a terminal and typing:

进入克隆下来的， maven的环境变量要先配好了

mvn install

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-l6Nv24Gf-1588124536236)(D:\笔记\13.毕业设计\5.emqx实践\1.基本教程.assets\image-20200409161110226.png)]

Copy the kafka-mqtt-source-connector-"version".jar from your maven target directory to the directory /usr/share/java/kafka:

sudo mkdir /usr/share/java/kafka
sudo cp ./target/*with-dependencies.jar /usr/share/java/kafka/.

sudo mkdir -p /usr/share/java/kafka
sudo cp ./target/*with-dependencies.jar /usr/share/java/kafka/.

Insecure - using tcp *Connect Standalone*

1. Uncomment plugin.path in "path-to-kafka"/kafka_2.13-2.4.1/config/connect-standalone.properties, so that it is set to

plugin.path=/usr/share/java,/usr/local/share/kafka/plugins,/usr/local/share/java/

vim /opt/software/kafka_2.13-2.4.1/config/connect-standalone.properties

vim $kafka_home/config/connect-standalone.properties

2.Copy the accompanying source connector properties file in this repository, source-connect-mqtt.properties, to "path-to-kafka"/kafka_2.13-2.4.1/config/ (or create a new properties file with the same name in the given directory).

复制文件到卡夫卡的config下面

cp /opt/software/kafka-mqtt-source-connector/src/main/resources/source-connect-mqtt.properties /opt/software/kafka_2.13-2.4.1/config/

cp /home/jiayuan/work/kafka-mqtt-source-connector/src/main/resources/source-connect-mqtt.properties  $kafka_home/config/

需要进行一下修改

rm  $kafka_home/config/source-connect-mqtt.properties  
subl  $kafka_home/config/source-connect-mqtt.properties  

3.Ensure the following configuration in source-connect-mqtt.properties:

name=mqtt-source-connector
tasks.max=1
connector.class=com.sintef.asam.MqttSourceConnector
mqtt.connector.broker.uri=tcp://0.0.0.0:1883
mqtt.connector.broker.topic=test/#
mqtt.conncetor.kafka.topic=test

where mqtt.connector.broker.topic sets the topic one wants to subscribe to in the mqtt broker, while mqtt.connector.kafka.topic sets the topic for publishing to the Kafka broker. The mqtt.connector.broker.uri needs to be set according to your own mqtt broker, but the default for mosquitto and emqx will be the abovementioned.

4.Start Connect Standalone with our connector by typing (this may take a minute or two):

"path-to-kafka"/kafka_2.13-2.4.1/bin/connect-standalone.sh "path-to-kafka"/kafka_2.13-2.4.1/config/connect-standalone.properties "path-to-kafka"/kafka_2.13-2.4.1/config/source-connect-mqtt.properties

/opt/software/kafka_2.13-2.4.1/bin/connect-standalone.sh /opt/software/kafka_2.13-2.4.1/config/connect-standalone.properties "path-to-kafka"/kafka_2.13-2.4.1/config/source-connect-mqtt.properties



$kafka_home/bin/connect-standalone.sh $kafka_home/config/connect-standalone.properties  $kafka_home/kafka_2.13-2.4.1/config/source-connect-mqtt.properties

*Connect Distributed* Kafka Connect Distributed不需要属性文件即可配置连接器。它使用Kafka Connect REST接口。

5.plugin.path在中取消注释"path-to-kafka"/kafka_2.13-2.4.1/config/connect-distributed.properties，以便将其设置为

vim /opt/software/kafka_2.13-2.4.1/config/connect-distributed.properties

subl   $kafka_home/config/connect-distributed.properties

plugin.path=/usr/share/java,/usr/local/share/kafka/plugins,/usr/local/share/java/

并将rest.port其设置为

rest.port=19005

这将有助于避免某些“绑定”异常。这将是Connect REST接口的端口。

6. 通过键入（可能需要一两分钟）来启动Connect Distributed with：

"path-to-kafka"/kafka_2.13-2.4.1/bin/connect-distributed.sh "path-to-kafka"/kafka_2.13-2.4.1/config/connect-distributed.properties

/opt/software/kafka_2.13-2.4.1/bin/connect-distributed.sh /opt/software/kafka_2.13-2.4.1/config/connect-distributed.properties


$kafka_home/bin/connect-distributed.sh  $kafka_home/config/connect-distributed.properties

7.通过将以下命令发布到Connect REST接口来启动我们的连接器：

curl -s -X POST -H 'Content-Type: application/json' http://127.0.0.1:19005/connectors -d '{"name":"mqtt-source-connector","config":{"connector.class":"com.sintef.asam.MqttSourceConnector","tasks.max":"1","mqtt.connector.broker.uri":"tcp://localhost:1883", "mqtt.connector.broker.topic":"test/#","mqtt.connector.kafka.topic":"test"}}'

8.检查启动Conncet Distributed的终端，并在连接器似乎已成功启动之后，通过键入以下命令检查是否存在：

curl 'Content-Type: application/json' http://127.0.0.1:19005/connectors

其中响应是一个带有名称的连接器的数组。

9.通过让Kafka Consumer订阅以下主题来测试连接器test：

这部分改成自己的消费者

Documents/confluent-5.4.0/bin/kafka-console-consumer --bootstrap-server 127.0.0.1:9092 --topic test

消费消息

 $kafka_home/bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --from-beginning --topic test

–from-beginning：会把主题中以往

10 然后使用mosquitto将一些内容发布到EMQX代理上的测试主题中：

mosquitto_pub -h 127.0.0.1 -p 1883 -t test -m "Hello, world"

并看到消息显示为base64编码到您的Kafka Consumer。