一、下载安装kafka
重装一套zookeeper集群和Kafka集群,这里选择的是Kafka集成了zookeeper的安装包：
下载地址：https://archive.apache.org/dist/kafka/2.8.1/kafka_2.12-2.8.1.tgz
安装参考：https://blog.csdn.net/Astralisxoxo/article/details/120415140
二、下载mysql（此处用docker）
docker run -p 3306:3306 --name mysql-01 \
-v /mydata/mysql-01/log:/var/log/mysql \
-v /mydata/mysql-01/data:/var/lib/mysql \
-v /mydata/mysql-01/conf:/etc/mysql/conf.d \
-e MYSQL_ROOT_PASSWORD=root \
--restart=always \
-d mysql:5.7
配置初始化
mkdir -p /mydata/mysql-01/conf && vi /mydata/mysql-01/conf/default.cnf
配置内容如下：
[mysqld]
#mysql开启binlog
server-id= 1
log-bin=mysql-bin
binlog_format=row
init_connect='SET collation_connection = utf8mb4_unicode_ci'
init_connect='SET NAMES utf8mb4'
character-set-server=utf8mb4
collation-server=utf8mb4_unicode_ci
skip-character-set-client-handshake
skip-name-resolve
进入容器：
docker exec -it 容器id /bin/bash
检测binlog是否开启成功:
mysql -uroot -proot -e "show variables like'%log_bin%'"
三、下载安装debezium-connector-mysql插件
wget https://repo1.maven.org/maven2/io/debezium/debezium-connector-mysql/1.7.1.Final/debezium-connector-mysql-1.7.1.Final-plugin.tar.gz
解压到自己创建的目录
mkdir -p /opt/debezium/connector
tar -zxvf debezium-connector-mysql-1.7.1.Final-plugin.tar.gz -C /opt/debezium/connector/
打开kafka配置文件connect-distributed.properties 进行配置
vim /opt/kafka-2.4.1/config/connect-distributed.properties
bootstrap.servers=192.168.1.200:9092
group.id=connect-mysql
key.converter=org.apache.kafka.connect.json.JsonConverter
value.converter=org.apache.kafka.connect.json.JsonConverter
key.converter.schemas.enable=false
value.converter.schemas.enable=false
status.storage.topic=connect-mysql-status
status.storage.replication.factor=1
offset.flush.interval.ms=10000
plugin.path=/opt/debezium/connector
四、启动各个组件
4.1启动mysql, zookeeper,kafka
4.2启动Kafka connector
bin/connect-distributed.sh -daemon config/connect-distributed.properties
4.3查看是否启动
jps -l
4.4.检测kafka连接器的服务状态
curl -H "Accept:application/json" 192.168.1.200:8083
4.5.检查向 Kafka Connect 注册的连接器列表
curl -H "Accept:application/json"192.168.1.200:8083/connectors/
返回空列表, 表示目前还没有注册的连接器
五、部署Debezium Mysql Connector
5.1配置信息
{
"name": "connector-test",
"config": {
"connector.class": "io.debezium.connector.mysql.MySqlConnector",
"database.hostname": "192.168.1.200",
"database.port": "3306",
"database.user": "root",
"database.password": "root",
"database.server.id": "10086",
"database.server.name": "bigdata",
"database.include.list": "bpm12",
"database.history.kafka.bootstrap.servers": "192.168.1.200:9092",
"database.history.kafka.topic": "schema-changes.inventory"
}
}
说明:
name: 连接器名字
database.include.list: 要监控的数据库列表
database.server.name: 服务器名. 会成为topic的前缀
5.2注册连接器
Kafka Connect 服务的 API 提交POST针对/connectors资源的请求，其中包含描述新连接器（称为inventory-connector）的 JSON 文档。
curl -i -X POST -H "Accept:application/json" -H "Content-Type:application/json" 192.168.1.200:8083/connectors/ -d '{"name":"connector-test","config":{"connector.class":"io.debezium.connector.mysql.MySqlConnector","database.hostname":"192.168.1.200","database.port":"3306","database.user":"root","database.password":"root","database.server.id":"10086","database.server.name":"bigdata","database.include.list":"bpm12","database.history.kafka.bootstrap.servers":"192.168.1.200:9092","database.history.kafka.topic":"schema-changes.inventory"}}'
检查连接器是否注册
curl -H "Accept:application/json"192.168.1.200:8083/connectors/
六、测试
在debezium启动后，会创建对应表的topic
6.1查看topic
bin/kafka-topics.sh --list --zookeeper 192.168.1.200:2181
6.2方便监控，开启消费者监听消息的变化
bin/kafka-console-consumer.sh --bootstrap-server 192.168.1.200:9092 --topic bigdata.bpm12.infomover_project --from-beginning
会发现消费者已经消费了一条消息:首次读取的数据：debezium会读取库中所有的表，然后生成快照
{
"before": null,
"after": {
"id": 16,
"apply_chinese_name": "管理员",
"create_time": 1652392038000,
"creator": "admin",
"creator_id": 1,
"current_approver": null,
"current_step": null,
"documentId": null,
"enabled": 1,
"item_describe": "test",
"name": "第一個測試項目",
"processId": null,
"taskId": null,
"update_time": 1652392038000
},
"source": {
"version": "1.7.1.Final",
"connector": "mysql",
"name": "bigdata",
"ts_ms": 1652429877866,
"snapshot": "true",
"db": "bpm12",
"sequence": null,
"table": "infomover_project",
"server_id": 0,
"gtid": null,
"file": "mysql-bin.000001",
"pos": 3892,
"row": 0,
"thread": null,
"query": null
},
"op": "r",
"ts_ms": 1652429877866,
"transaction": null
}
向对应的数据库表增加数据,监听到的数据变化为:
{
"before": null,
"after": {
"id": 20,
"apply_chinese_name": "管理员",
"create_time": 1652463906000,
"creator": "admin",
"creator_id": 1,
"current_approver": null,
"current_step": null,
"documentId": null,
"enabled": 1,
"item_describe": "测试kafkaConnect与debezium mysql connector",
"name": "测试kafkaConnect",
"processId": null,
"taskId": null,
"update_time": 1652463906000
},
"source": {
"version": "1.7.1.Final",
"connector": "mysql",
"name": "bigdata",
"ts_ms": 1652435106000,
"snapshot": "false",
"db": "bpm12",
"sequence": null,
"table": "infomover_project",
"server_id": 1,
"gtid": null,
"file": "mysql-bin.000001",
"pos": 8114,
"row": 0,
"thread": null,
"query": null
},
"op": "c",
"ts_ms": 1652435106624,
"transaction": null
}
向对应的数据库表修改数据，监听到的变化为:
{
"before": {
"id": 20,
"apply_chinese_name": "管理员",
"create_time": 1652463906000,
"creator": "admin",
"creator_id": 1,
"current_approver": null,
"current_step": null,
"documentId": null,
"enabled": 1,
"item_describe": "测试kafkaConnect与debezium mysql connector",
"name": "测试kafkaConnect",
"processId": null,
"taskId": null,
"update_time": 1652463906000
},
"after": {
"id": 20,
"apply_chinese_name": "管理员",
"create_time": 1652463906000,
"creator": "admin",
"creator_id": 1,
"current_approver": null,
"current_step": null,
"documentId": null,
"enabled": 1,
"item_describe": "测试kafkaConnect与debezium mysql connector的修时的采集",
"name": "测试kafkaConnect",
"processId": null,
"taskId": null,
"update_time": 1652464149000
},
"source": {
"version": "1.7.1.Final",
"connector": "mysql",
"name": "bigdata",
"ts_ms": 1652435348000,
"snapshot": "false",
"db": "bpm12",
"sequence": null,
"table": "infomover_project",
"server_id": 1,
"gtid": null,
"file": "mysql-bin.000001",
"pos": 10607,
"row": 0,
"thread": null,
"query": null
},
"op": "u",
"ts_ms": 1652435349030,
"transaction": null
}
删除数据得采集为：
{
"before": {
"id": 20,
"apply_chinese_name": "管理员",
"create_time": 1652463906000,
"creator": "admin",
"creator_id": 1,
"current_approver": null,
"current_step": null,
"documentId": null,
"enabled": 1,
"item_describe": "测试kafkaConnect与debezium mysql connector的修时的采集",
"name": "测试kafkaConnect",
"processId": null,
"taskId": null,
"update_time": 1652464149000
},
"after": null,
"source": {
"version": "1.7.1.Final",
"connector": "mysql",
"name": "bigdata",
"ts_ms": 1652435423000,
"snapshot": "false",
"db": "bpm12",
"sequence": null,
"table": "infomover_project",
"server_id": 1,
"gtid": null,
"file": "mysql-bin.000001",
"pos": 11957,
"row": 0,
"thread": null,
"query": null
},
"op": "d",
"ts_ms": 1652435423212,
"transaction": null
}
总体采集如图：
947
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
