flink-SQL变更日志打印

最新推荐文章于 2024-08-19 11:43:11 发布

重生之我是一名程序员

最新推荐文章于 2024-08-19 11:43:11 发布

阅读量529

点赞数

分类专栏： flink 文章标签： flink

原文链接：https://editor.csdn.net/md/?not_checkout=1&spm=1001.2101.3001.5352

版权

flink 专栏收录该内容

10 篇文章 0 订阅

订阅专栏

该代码示例展示了如何在Flink1.15.4中使用SQL处理MongoDB的变更数据捕获(CDC)。通过创建数据源表并定义MongoDB连接器，程序从MongoDB的t_article_source集合读取变更事件，并将其打印到控制台。日志配置文件(log4j2.xml)设定了日志输出级别为INFO，用于记录操作日志。

摘要由CSDN通过智能技术生成

flink-SQL变更日志打印

基于flink-1.15.4

很多时候会进行 SQL 变量调试

public class MongoDbEmbededPrinterCdc {


    /**
     * flink-cdc-mongo https://ververica.github.io/flink-cdc-connectors/master/content/connectors/mongodb-cdc.html#data-type-mapping
     * https://github.com/ververica/flink-cdc-connectors/blob/master/docs/content/connectors/mongodb-cdc(ZH).md
     * @param args
     * @throws Exception
     */
    public static void main(String[] args) throws Exception {

        StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
        StreamTableEnvironment tableEnv = StreamTableEnvironment.create(env);
        // 数据源表
        String sourceDDL =
               "CREATE TABLE `t_article_source` (\n" +
                "  `_id` STRING,\n" +
                "  `articleId` INT,\n" +
                "  `title` STRING,\n" +
                "  `tags` ARRAY<String>,\n"+
                "  `authors` ARRAY<ROW<name STRING, address STRING>>,\n"+
                "  PRIMARY KEY (`_id`) NOT ENFORCED\n" + //mongo 主键 Primary key must be _id field
                ") WITH (\n" +
                        " 'connector' = 'mongodb-cdc',\n" +
                        " 'hosts' = '172.1x.0.xx:27017',\n" + //多个 host
                        " 'username' = 'flinkuser',\n" +
                        " 'password' = 'flinkpw',\n" +
                        " 'database' = 'test',\n" +
                        " 'collection' = 't_article_source'\n" +
                        ")";
        
        tableEnv.executeSql(sourceDDL);
        Table table = tableEnv.from("t_article_source");
        tableEnv.toChangelogStream(table).print("mongo");
        env.execute("abc");
    }

}

log4j2.xml 日志:

<?xml version="1.0" encoding="UTF-8"?>
<configuration monitorInterval="5">
    <Properties>
        <property name="LOG_PATTERN" value="%date{HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %msg%n"/>
        <!-- LOG_LEVEL 配置你需要的日志输出级别       -->
        <property name="LOG_LEVEL" value="INFO"/>
    </Properties>

    <appenders>
        <console name="Console" target="SYSTEM_OUT">
            <PatternLayout pattern="${LOG_PATTERN}"/>
            <ThresholdFilter level="${LOG_LEVEL}" onMatch="ACCEPT" onMismatch="DENY"/>
        </console>
    </appenders>

    <loggers>
        <root level="${LOG_LEVEL}">
            <appender-ref ref="Console"/>
        </root>
    </loggers>

</configuration>

输出:

mongo:14> +I[649d6ed7084687af5e575091, 3, 小马过河2, [玄幻, 修真2], [+I[张三2, 上海2]]]
mongo:9> -U[649cea151df115e06dd24990, 1, 标1题3, [玄幻, 修真2], [+I[张三2, 上海2]]]
mongo:9> +U[649cea151df115e06dd24990, 1, 标1题31, [玄幻, 修真2], [+I[张三2, 上海2]]]