flink sql多表join

本文介绍如何使用 Apache Flink 的 SQL 功能从 Kafka 消费数据,并将多个表的数据整合后同步到 ClickHouse 中。具体包括定义 Kafka 和 ClickHouse 的连接配置、创建源表和目标表以及编写 SQL 查询来实现数据的实时处理与同步。
摘要由CSDN通过智能技术生成

val createTable =
  """
    |CREATE TABLE nt_sale_order (
    |    id VARCHAR,
    |    write_date BIGINT,
    |    create_uid INT,
    |    name VARCHAR,
    |    op VARCHAR
    |)
    |WITH (
    |    'connector' = 'kafka',
    |    'topic' = 'shopforce.public.nt_sale_order',
    |    'scan.startup.mode' = 'latest-offset',
    |    'properties.bootstrap.servers' = '192.168.10.16:9092',
    |    'properties.group.id' = 'testGroup',
    |    'format' = 'debezium-json',
    |    'debezium-json.schema-include' = 'true'
    |)
  """.stripMargin

val createTable1 =
  """
    |CREATE TABLE nt_sale_order_line (
    |    id VARCHAR,
    |    order_id VARCHAR,
    |    unit_price VARCHAR,
    |    op VARCHAR
    |)
    |WITH (
    |    'connector' = 'kafka',
    |    'topic' = 'shopforce.public.nt_sale_order_line',
    |    'scan.startup.mode' = 'latest-offset',
    |    'properties.bootstrap.servers' = '192.168.10.16:9092',
    |    'properties.group.id' = 'testGroup1',
    |    'format' = 'debezium-json',
    |    'debezium-json.schema-include' = 'true'
    |)
  """.stripMargin


val createTable2 =
  """
    |CREATE TABLE nt_order_payment (
    |    id VARCHAR,
    |    order_id VARCHAR,
    |    name VARCHAR,
    |    paid_at VARCHAR,
    |    op VARCHAR
    |)
    |WITH (
    |    'connector' = 'kafka',
    |    'topic' = 'shopforce.public.nt_payment',
    |    'scan.startup.mode' = 'latest-offset',
    |    'properties.bootstrap.servers' = '192.168.10.16:9092',
    |    'properties.group.id' = 'testGroup1',
    |    'format' = 'debezium-json',
    |    'debezium-json.schema-include' = 'true'
    |)
  """.stripMargin

tblEnv.executeSql(createTable)
tblEnv.executeSql(createTable1)
tblEnv.executeSql(createTable2)


val nt_order_detail =
  """
    |CREATE TABLE nt_order_detail(
    |id VARCHAR primary key,
    |name VARCHAR,
    |unit_price VARCHAR,
    |paid_at VARCHAR
    |)
    |WITH(
    |'connector' = 'jdbc',
    |'url' = 'jdbc:clickhouse://192.168.10.16:8123/default',
    |'table-name' = 'nt_order_detail'
    |)
  """.stripMargin

tblEnv.executeSql(nt_order_detail)


var sqlQuery =
  """
    |insert into nt_order_detail
    |SELECT id,name,unit_price,paid_at from(
    |SELECT p.id,b.name,p.unit_price,s.paid_at from nt_sale_order_line AS p
        |JOIN nt_sale_order AS b
    |  ON b.id = p.order_id
        |JOIN nt_order_payment AS s
    |  ON b.id = s.order_id where b.op = 'c' and p.op = 'c' and s.paid_at is not null)a
    |""".stripMargin

tblEnv.executeSql(sqlQuery)

 

评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值