CDC变化数据捕获——Debezium-Embedded

52 篇文章 7 订阅
14 篇文章 0 订阅
本文详细介绍了如何配置MySQL主从同步并使用Debezium-Embedded在Java应用中捕获MySQL数据库的变化数据。通过示例代码展示了如何创建和使用DebeziumEngine,包括简单的数据通知、使用Connect类创建引擎以及批量处理CDC事件。适用于无需Kafka的场景,便于直接获取和处理数据库变更事件。
摘要由CSDN通过智能技术生成

CDC变化数据捕获——Debezium-Embedded

0. 前言

  • CDC(Change Data Capture)是变化数据捕获的意思,可以捕获数据库数据的增加、更新、删除等记录,RedHat的 Debezium 正是这样一款产品。
  • 对于数据库数据变化的监听,国内常用的一般是阿里的 Canal ,可惜目前仅支持MySQL。而相对来说,Debezium支持的数据库就非常丰富了,包括:MySQL、PostgreSQL、MongoDB、Oracle、SQL Server、Db2、Cassandra、Vitess。
  • 使用Debezium比较麻烦的是,它通常需要用到 Apache Kafka 来保证数据的高容错性和高可靠性。但是,很多应用不需要这样的特性,或者不想连接Kafka,因此Debezium提供了debezium-embedded。利用它,可以在Java应用中连接到数据库,直接获取到数据库的CDC事件信息,不用额外进行其他操作。
  • debezium-embedded使用起来更为便捷、灵巧,目前国内已有阿里的Flink-CDC在使用它了(查看示例)。可惜的是Debezium在国内相对来说文档不多,其嵌入式使用方式的示例更是难找(官方英文文档有坑- -!!),本文的目的正是要写出此处的示例(MySQL),方便大家参阅。
  • 官方文档地址: Debezium Engine

1. 配置MySQL主从同步

  • 停止MySQL服务
  • 修改配置文件,在[mysqld]处添加内容如下
    # 一般情况下,Window修改mysql目录下的my.ini
    # 一般情况下,Linux修改/etc/mysql下的my.cnf
    
    [mysqld]
    # 添加的部分,server-id随便填
    server-id = 12345
    log-bin = mysql-bin
    # 必须为ROW
    binlog_format = ROW
    # 必须为FULL,MySQL-5.7后才有该参数
    binlog_row_image  = FULL
    expire_logs_days  = 10
    
  • 启动MySQL服务
  • root 登入MySQL,查看binlog相关变量配置
    SHOW VARIABLES LIKE '%binlog%';
    
  • root 登入MySQL,新增用于同步数据的用户
    -- 设置拥有同步权限的用户
    CREATE USER 'cdc_user' IDENTIFIED BY 'cdc_password';
    -- 赋予同步相关权限
    GRANT SELECT, RELOAD, SHOW DATABASES, REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO 'cdc_user';
    
  • 新用户 cdc_user 登入MySQL,查看主从同步情况
    SHOW MASTER STATUS;
    SHOW SLAVE STATUS;
    SHOW BINARY LOGS;
    
  • 建议再建个测试用的库和表(带主键),方便调试,例如Flink-CDC中的

2. Debezium-Embedded 代码开发

2.1 Maven导包

    <properties>
        <maven.compiler.source>8</maven.compiler.source>
        <maven.compiler.target>8</maven.compiler.target>
        <version.debezium>1.4.2.Final</version.debezium>
    </properties>

    <dependencies>
        <dependency>
            <groupId>io.debezium</groupId>
            <artifactId>debezium-api</artifactId>
            <version>${version.debezium}</version>
        </dependency>
        <dependency>
            <groupId>io.debezium</groupId>
            <artifactId>debezium-embedded</artifactId>
            <version>${version.debezium}</version>
        </dependency>
        <dependency>
            <groupId>io.debezium</groupId>
            <artifactId>debezium-connector-mysql</artifactId>
            <version>${version.debezium}</version>
        </dependency>
    </dependencies>

2.2 代码-简单示例

import io.debezium.connector.mysql.MySqlConnector;
import io.debezium.engine.ChangeEvent;
import io.debezium.engine.DebeziumEngine;
import io.debezium.engine.format.Json;
import io.debezium.relational.history.FileDatabaseHistory;
import org.apache.kafka.connect.storage.FileOffsetBackingStore;

import java.util.Properties;
import java.util.UUID;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;

public class Demo01 {

    public static void main(String[] args) {
        // 0. 配置数据库,添加用户,赋予主从同步的权限

        // 1. 生成配置
        Properties props = genProps();

        // 2. 构建 DebeziumEngine
        // 使用 Json 格式
        DebeziumEngine<ChangeEvent<String, String>> engine = DebeziumEngine.create(Json.class)
                .using(props)
                .notifying(record -> {
                	// record中会有操作的类型(增、删、改)和具体的数据
                	// key是主键
                    System.out.println("record.key() = " + record.key());
                    System.out.println("record.value() = " + record.value());
                })
                .using((success, message, error) -> {
               		// 强烈建议加上此部分的回调代码,方便查看错误信息
                    
                    if (!success && error != null) {
                    	// 报错回调
                        System.out.println("----------error------");
                        System.out.println(message);
                    }
                }).build();

        // 3. 正式运行
        ExecutorService executor = Executors.newSingleThreadExecutor();
        executor.execute(engine);
    }

    private static Properties genProps() {
        // 配置
        Properties props = new Properties();
        // 在maven处引入其他数据库的连接器,例如debezium-connector-postgres,再修改此处的connector.class,即可使用其他数据库的CDC
        props.setProperty("connector.class", MySqlConnector.class.getCanonicalName());
        props.setProperty("database.server.name", "my_server_01"); // 可以任意修改
        props.setProperty("database.hostname", "localhost"); // IP
        props.setProperty("database.port", String.valueOf(3306)); // 端口
        props.setProperty("database.user", "cdc_user"); // 用户
        props.setProperty("database.password", "cdc_password"); // 密码
        props.setProperty("database.serverTimezone", "UTC"); // 时区
        // 下面两个是数据库和表,注意只能选择一种:
        // 1. 使用database.whitelist,只设置数据库(会通知全库的CDC信息)
        // 2. 使用table.whitelist,设置库名和表名(会通知单个库的单个表的CDC信息)
//        props.setProperty("database.whitelist", "db_inventory_cdc");
        props.setProperty("table.whitelist", "db_inventory_cdc.tb_products_cdc"); // 库.表名

        props.setProperty("name", "engine");
        props.setProperty("key.converter.schemas.enable", "false");
        props.setProperty("value.converter.schemas.enable", "false");
        props.setProperty("include.schema.changes", "false");
        props.setProperty("tombstones.on.delete", "false");
        props.setProperty("database.history", FileDatabaseHistory.class.getCanonicalName());
        props.setProperty("database.history.store.only.monitored.tables.ddl", "true");
        props.setProperty("database.history.file.filename", "./storage/dbhistory.dat");
        props.setProperty("database.history.instance.name", UUID.randomUUID().toString());
        props.setProperty("database.history.skip.unparseable.ddl", "true");
        // 偏移量持久化配置
		props.setProperty("offset.storage", FileOffsetBackingStore.class.getCanonicalName());
        props.setProperty("offset.storage.file.filename", "./tmp/offsets.dat");
        props.setProperty("offset.flush.interval.ms", "6000");
        return props;
    }
    
}

2.3 代码-使用Connect.class创建引擎

import io.debezium.embedded.Connect;

……

DebeziumEngine<ChangeEvent<SourceRecord, SourceRecord>> engine = DebeziumEngine.create(Connect.class)
        .using(props)
        .notifying(record -> {
              System.out.println("record.key = " + record.key());
              System.out.println("record.value = " + record.value());
              System.out.println("record.value.sourcePartition = " + record.value().sourcePartition());
              System.out.println("record.value.sourceOffset = " + record.value().sourceOffset());
              System.out.println("record.value.key = " + record.value().key());
              System.out.println("record.value.value = " + record.value().value());
              System.out.println("record.destination = " + record.destination());
          })
          .using((success, message, error) -> {
              // 省略代码
          }).build();

2.4 代码-批量处理CDC事件

DebeziumEngine<ChangeEvent<String, String>> engine = DebeziumEngine.create(Json.class)
        .using(props)
        .notifying((records, committer) -> {
        	// records是当前批次的数据
            for (ChangeEvent<String, String> record : records) {
                System.out.println("record.key() = " + record.key());
                System.out.println("record.value() = " + record.value());
            }
            // 当前批次处理完后,需要标记为完成
            committer.markBatchFinished();
        })
        .using((success, message, error) -> {
            // 省略代码
        }).build();

2.5 启动代码运行即可

  • 控制台日志样例
record.key() = {"id":1}
record.value() = {"before":null,"after":{"id":1,"name":"xiaowang","age":34,"address":"beijing"},"source":{"version":"1.4.2.Final","connector":"mysql","name":"my_server_01","ts_ms":0,"snapshot":"true","db":"y1","table":"tb_test_from","server_id":0,"gtid":null,"file":"binlog.000030","pos":115750444,"row":0,"thread":null,"query":null},"op":"r","ts_ms":1679554873026,"transaction":null}
record.key() = {"id":2}
record.value() = {"before":null,"after":{"id":2,"name":"lilei","age":12,"address":"chongqing"},"source":{"version":"1.4.2.Final","connector":"mysql","name":"my_server_01","ts_ms":0,"snapshot":"true","db":"y1","table":"tb_test_from","server_id":0,"gtid":null,"file":"binlog.000030","pos":115750444,"row":0,"thread":null,"query":null},"op":"r","ts_ms":1679554873026,"transaction":null}
record.key() = {"id":3}
record.value() = {"before":null,"after":{"id":3,"name":"meimei","age":19,"address":"chengdu"},"source":{"version":"1.4.2.Final","connector":"mysql","name":"my_server_01","ts_ms":0,"snapshot":"true","db":"y1","table":"tb_test_from","server_id":0,"gtid":null,"file":"binlog.000030","pos":115750444,"row":0,"thread":null,"query":null},"op":"r","ts_ms":1679554873026,"transaction":null}
record.key() = {"id":4}
record.value() = {"before":null,"after":{"id":4,"name":"chunfang","age":18,"address":"chongqing"},"source":{"version":"1.4.2.Final","connector":"mysql","name":"my_server_01","ts_ms":0,"snapshot":"true","db":"y1","table":"tb_test_from","server_id":0,"gtid":null,"file":"binlog.000030","pos":115750444,"row":0,"thread":null,"query":null},"op":"r","ts_ms":1679554873026,"transaction":null}
record.key() = {"id":6}
record.value() = {"before":null,"after":{"id":6,"name":"lili","age":15,"address":"chongqing"},"source":{"version":"1.4.2.Final","connector":"mysql","name":"my_server_01","ts_ms":0,"snapshot":"true","db":"y1","table":"tb_test_from","server_id":0,"gtid":null,"file":"binlog.000030","pos":115750444,"row":0,"thread":null,"query":null},"op":"r","ts_ms":1679554873026,"transaction":null}
record.key() = {"id":7}
record.value() = {"before":null,"after":{"id":7,"name":"xiaoming","age":21,"address":"beijing"},"source":{"version":"1.4.2.Final","connector":"mysql","name":"my_server_01","ts_ms":0,"snapshot":"true","db":"y1","table":"tb_test_from","server_id":0,"gtid":null,"file":"binlog.000030","pos":115750444,"row":0,"thread":null,"query":null},"op":"r","ts_ms":1679554873026,"transaction":null}
record.key() = {"id":8}
record.value() = {"before":null,"after":{"id":8,"name":"wangwu","age":24,"address":"hunan"},"source":{"version":"1.4.2.Final","connector":"mysql","name":"my_server_01","ts_ms":0,"snapshot":"true","db":"y1","table":"tb_test_from","server_id":0,"gtid":null,"file":"binlog.000030","pos":115750444,"row":0,"thread":null,"query":null},"op":"r","ts_ms":1679554873026,"transaction":null}
record.key() = {"id":9}
record.value() = {"before":null,"after":{"id":9,"name":"tom","age":12,"address":"beijing"},"source":{"version":"1.4.2.Final","connector":"mysql","name":"my_server_01","ts_ms":0,"snapshot":"true","db":"y1","table":"tb_test_from","server_id":0,"gtid":null,"file":"binlog.000030","pos":115750444,"row":0,"thread":null,"query":null},"op":"r","ts_ms":1679554873026,"transaction":null}
record.key() = {"id":10}
record.value() = {"before":null,"after":{"id":10,"name":"whitee","age":55,"address":"hebei"},"source":{"version":"1.4.2.Final","connector":"mysql","name":"my_server_01","ts_ms":0,"snapshot":"true","db":"y1","table":"tb_test_from","server_id":0,"gtid":null,"file":"binlog.000030","pos":115750444,"row":0,"thread":null,"query":null},"op":"r","ts_ms":1679554873026,"transaction":null}
record.key() = {"id":11}
record.value() = {"before":null,"after":{"id":11,"name":"meimei","age":31,"address":"hebei"},"source":{"version":"1.4.2.Final","connector":"mysql","name":"my_server_01","ts_ms":0,"snapshot":"true","db":"y1","table":"tb_test_from","server_id":0,"gtid":null,"file":"binlog.000030","pos":115750444,"row":0,"thread":null,"query":null},"op":"r","ts_ms":1679554873026,"transaction":null}
record.key() = {"id":12}
record.value() = {"before":null,"after":{"id":12,"name":"hehe","age":13,"address":"nanjing"},"source":{"version":"1.4.2.Final","connector":"mysql","name":"my_server_01","ts_ms":0,"snapshot":"true","db":"y1","table":"tb_test_from","server_id":0,"gtid":null,"file":"binlog.000030","pos":115750444,"row":0,"thread":null,"query":null},"op":"r","ts_ms":1679554873026,"transaction":null}
record.key() = {"id":13}
record.value() = {"before":null,"after":{"id":13,"name":"liyang","age":16,"address":"beijing"},"source":{"version":"1.4.2.Final","connector":"mysql","name":"my_server_01","ts_ms":0,"snapshot":"true","db":"y1","table":"tb_test_from","server_id":0,"gtid":null,"file":"binlog.000030","pos":115750444,"row":0,"thread":null,"query":null},"op":"r","ts_ms":1679554873026,"transaction":null}
record.key() = {"id":14}
record.value() = {"before":null,"after":{"id":14,"name":"huahua","age":21,"address":"shandong"},"source":{"version":"1.4.2.Final","connector":"mysql","name":"my_server_01","ts_ms":0,"snapshot":"true","db":"y1","table":"tb_test_from","server_id":0,"gtid":null,"file":"binlog.000030","pos":115750444,"row":0,"thread":null,"query":null},"op":"r","ts_ms":1679554873026,"transaction":null}
record.key() = {"id":15}
record.value() = {"before":null,"after":{"id":15,"name":"zhangsan","age":25,"address":"hubei"},"source":{"version":"1.4.2.Final","connector":"mysql","name":"my_server_01","ts_ms":0,"snapshot":"last","db":"y1","table":"tb_test_from","server_id":0,"gtid":null,"file":"binlog.000030","pos":115750444,"row":0,"thread":null,"query":null},"op":"r","ts_ms":1679554873026,"transaction":null}
// 删除了一条带IP的日志
record.key() = {"id":14}
record.value() = {"before":{"id":14,"name":"huahua","age":21,"address":"shandong"},"after":{"id":14,"name":"huahua","age":10,"address":"shandong"},"source":{"version":"1.4.2.Final","connector":"mysql","name":"my_server_01","ts_ms":1679554893000,"snapshot":"false","db":"y1","table":"tb_test_from","server_id":1,"gtid":null,"file":"binlog.000030","pos":115750677,"row":0,"thread":26773,"query":null},"op":"u","ts_ms":1679554893186,"transaction":null}
record.key() = {"id":16}
record.value() = {"before":null,"after":{"id":16,"name":"alice","age":18,"address":"dalian"},"source":{"version":"1.4.2.Final","connector":"mysql","name":"my_server_01","ts_ms":1679554929000,"snapshot":"false","db":"y1","table":"tb_test_from","server_id":1,"gtid":null,"file":"binlog.000030","pos":115752466,"row":0,"thread":26773,"query":null},"op":"c","ts_ms":1679554929396,"transaction":null}
record.key() = {"id":6}
record.value() = {"before":{"id":6,"name":"lili","age":15,"address":"chongqing"},"after":null,"source":{"version":"1.4.2.Final","connector":"mysql","name":"my_server_01","ts_ms":1679554936000,"snapshot":"false","db":"y1","table":"tb_test_from","server_id":1,"gtid":null,"file":"binlog.000030","pos":115752780,"row":0,"thread":26773,"query":null},"op":"d","ts_ms":1679554936366,"transaction":null}
### 回答1: Flutter是Google开源的一款跨平台的UI开发框架,可以快速地构建高质量的移动、Web和桌面应用程序。而Embedded Linux是指Linux操作系统在嵌入式设备中的应用。通过将Flutter与Embedded Linux结合起来,可以让开发者在嵌入式设备上创建漂亮且功能强大的应用程序。 Flutter的跨平台特性使得开发者可以使用相同的代码库在多个平台上运行应用程序,包括嵌入式设备。嵌入式设备通常资源有限,但Flutter的轻量级和高性能使得它非常适合在此类设备上运行。Flutter的热重载功能可以让开发者快速迭代和测试应用程序,提高开发效率。 使用Flutter-Embedded Linux,开发者可以利用Flutter的丰富的UI组件和强大的渲染引擎来创建嵌入式设备上的用户界面。Flutter提供了丰富的UI控件和动画效果,可以实现各种视觉效果和交互体验。Flutter还具备良好的跨平台兼容性和高度定制性,可以适应不同嵌入式设备的各种需求。 除此之外,Flutter-Embedded Linux还提供了与Linux操作系统的良好集成,可以调用底层系统资源和API,如文件系统、网络接口等。这使得开发者可以通过Flutter在嵌入式设备上实现更强大的功能,例如与传感器的交互、与云服务的连接等。 总而言之,Flutter-Embedded Linux是一个强大的开发工具,可以帮助开发者在嵌入式设备中创建高质量的应用程序。它结合了Flutter的跨平台特性和Embedded Linux的优势,为开发者提供了更灵活、高效的开发环境。无论是工业控制、智能家居还是其他嵌入式应用,Flutter-Embedded Linux都能提供良好的开发体验和用户体验。 ### 回答2: Flutter-embedded-linux是指在嵌入式Linux平台上使用Flutter框架进行应用开发的一种技术方案。嵌入式Linux是指在资源有限的嵌入式设备上运行Linux操作系统的系统;而Flutter是一种跨平台的移动应用开发框架,可以同时在Android和iOS上运行。 使用Flutter-embedded-linux可以让开发者在嵌入式Linux平台上开发出跨平台的应用。这样一来,开发者不再需要针对不同的平台编写不同的代码,大大提高了开发效率。同时,Flutter的高性能和优秀的用户体验也被保留在嵌入式设备上。 在使用Flutter-embedded-linux时,开发者需要在嵌入式Linux平台上搭建Flutter的开发环境,包括安装Flutter SDK、配置相关的依赖库等。然后,可以使用Flutter提供的开发工具和API进行应用的开发和调试。 Flutter-embedded-linux可以应用于很多领域,例如智能家居、智能工业设备、嵌入式系统等。开发者可以利用Flutter-embedded-linux开发各种类型的应用,如物联网设备控制应用、工业监控系统、嵌入式音视频播放器等。 需要注意的是,由于嵌入式Linux平台的资源限制,开发者在使用Flutter-embedded-linux时需要特别关注应用的性能和资源消耗情况,避免出现过多的资源占用或性能瓶颈。 总之,Flutter-embedded-linux为开发者在嵌入式Linux平台上开发跨平台应用提供了一种方便、高效、高性能的技术方案,有望在嵌入式设备领域得到广泛应用。 ### 回答3: Flutter-Embedded-Linux是一种使用Flutter框架进行嵌入式开发的技术方案。Flutter是一种跨平台的开源UI框架,可以快速构建高性能、精美的应用程序。而Embedded-Linux则是嵌入式设备常用的操作系统。 通过将Flutter应用程序集成到Linux嵌入式系统中,可以在嵌入式设备上实现高效、流畅和美观的用户界面。Flutter的特点是能够在不同平台上保持一致的用户体验,因此在嵌入式系统上也能够实现类似于移动设备和桌面系统上的应用程序。 使用Flutter-Embedded-Linux进行嵌入式开发有一些优势。首先,Flutter具有快速构建UI界面的能力,可以节省开发时间。其次,Flutter对于动画和渲染效果的支持非常好,可以在嵌入式设备上实现更加流畅和绚丽的动画效果。此外,Flutter还支持热重载功能,可以实时更新应用程序而无需重新启动,极大地提高了开发效率。 然而,使用Flutter-Embedded-Linux也存在一些挑战。嵌入式设备通常资源有限,对占用内存和CPU的要求较高,需要对Flutter应用程序进行优化以提高性能。另外,由于Flutter是基于Dart语言开发的,需要事先熟悉Dart开发语言和Flutter框架的使用。 总之,Flutter-Embedded-Linux是一种用于嵌入式开发的技术方案,可以帮助开发者快速构建高品质的应用程序界面。它在嵌入式设备上的应用可以提供与移动设备和桌面系统相媲美的用户体验。但是,开发者需要注意对资源的合理利用和性能的优化,以确保应用程序在嵌入式设备上能够运行流畅。
评论 22
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值