在现有的kafka上,搭建debezium,并接收数据

本文档介绍了如何在已有Zookeeper和Kafka的环境中,通过Docker来部署Debezium,从MySQL数据库捕获binlog并将其发送到Kafka。首先,配置docker-compose.yml文件,启动连接器服务;然后,通过curl命令注册MySQL链接器,并指定相关配置如数据库信息、历史记录和主题;最后,检查和删除连接器状态,确保数据流正常运行。
摘要由CSDN通过智能技术生成

环境概要:

以有zookeeper、kafka,需要搭建一个debezium,接收mysql binlog,发送到kafka中。

开始!

docker-compose 没有安装的,自行百度

搭建debezium
cat docker-compose.yml

version: "3"
services:
  connect:
    image: debezium/connect:1.4
    ports:
      - 8083:8083
    environment:
      - BOOTSTRAP_SERVERS=kafkahost:port
      - GROUP_ID=1
      - CONFIG_STORAGE_TOPIC=debezium_connect_configs
      - OFFSET_STORAGE_TOPIC=debezium_connect_offsets
      - STATUS_STORAGE_TOPIC=debezium_connect_statuses
      - CONNECT_KEY_CONVERTER_SCHEMAS_ENABLE=false
      - CONNECT_VALUE_CONVERTER_SCHEMAS_ENABLE=false

docker-compose up -d
注册链接器
curl -i -X POST -H "Accept:application/json" -H  "Content-Type:application/json" http://localhost:8083/connectors/ -d @register-mysql11.json
cat register-mysql11.json
{
  "name": "mysql-kp-thailand-connector1",
  "config": {
    "connector.class": "io.debezium.connector.mysql.MySqlConnector",
    "tasks.max": "1",
    "tombstones.on.delete":"false",
    "database.hostname": "host",
    "database.port": "port",
    "database.user": "user",
    "database.password": "password",
    "database.server.id": "2107141",
    "database.server.name": "dbserver1",
    "database.include.list": "thailand_avatar",
    "table.include.list": "thailand_avatar.loan_order,thailand_avatar.loan_order_202102,thailand_avatar.loan_order_202103,thailand_avatar.loan_order_202104,thailand_avatar.loan_order_change_log,thailand_avatar.loan_order_not_white_user,thailand_avatar.loan_order_partial_payment,thailand_avatar.loan_order_rollover",
    "database.history.kafka.bootstrap.servers": "host:port",
    "database.history.kafka.topic": "schema-changes.thailand-cdc",
    "decimal.handling.mode": "string"
  }
}
查看链接器
curl -H "Accept:application/json" localhost:8083/connectors/
删除链接器
curl -X DELETE http://localhost:8083/connectors/inventory-connector2
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值