Flink实时仓库-DWD层(交易域-加购维度退化处理)模板代码

最新推荐文章于 2023-04-12 11:37:15 发布
最新推荐文章于 2023-04-12 11:37:15 发布
阅读量444 收藏 1
点赞数 1
分类专栏： 实时数据仓库 文章标签： flink java mybatis
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/S1124654/article/details/126031347
版权

简介

这里的代码就是双流join的经验

前置知识

intervalJoin

 

public class DoubleJoin {
    public static void main(String[] args) throws Exception {
//得到执行环境
        StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();


        DataStreamSource<String> socketTextStream = env.socketTextStream("master", 9999);

        //并行度设置为1才能看到效果，因为如果不为1，那么有些分区的水位线就是负无穷
        //由于自己的水位线是分区里面最小的水位线，那么自己的一直都是负无穷
        //就触发不了水位线的上升
        env.setParallelism(1);

        //第一个参数就一个名字，第二个参数用来表示事件时间
        SingleOutputStreamOperator<Tuple2<String, Long>> initData = socketTextStream.map(new MapFunction<String, Tuple2<String, Long>>() {
            @Override
            public Tuple2<String, Long> map(String value) throws Exception {
                String[] s = value.split(" ");
                //假设我们在控制台输入的参数是a 15s,那么我们要15*1000才能得到时间戳的毫秒时间
                return Tuple2.of(s[0], Long.parseLong(s[1]) * 1000L);
            }
        });

        //设置水位线
        SingleOutputStreamOperator<Tuple2<String, Long>> watermarks = initData.assignTimestampsAndWatermarks(
                // 针对乱序流插入水位线，延迟时间设置为 2s
                WatermarkStrategy.<Tuple2<String, Long>>forBoundedOutOfOrderness(Duration.ofSeconds(0))
                        .withTimestampAssigner(new SerializableTimestampAssigner<Tuple2<String, Long>>() {
                            @Override
                            public long extractTimestamp(Tuple2<String, Long> element, long recordTimestamp) {
                                //指定事件时间
                                return element.f1;
                            }
                        })
        );


        DataStreamSource<String> socketTextStream2 = env.socketTextStream("master", 9998);

        //并行度设置为1才能看到效果，因为如果不为1，那么有些分区的水位线就是负无穷
        //由于自己的水位线是分区里面最小的水位线，那么自己的一直都是负无穷
        //就触发不了水位线的上升
        env.setParallelism(1);

        //第一个参数就一个名字，第二个参数用来表示事件时间
        SingleOutputStreamOperator<Tuple2<String, Long>> initData2 = socketTextStream2.map(new MapFunction<String, Tuple2<String, Long>>() {
            @Override
            public Tuple2<String, Long> map(String value) throws Exception {
                String[] s = value.split(" ");
                //假设我们在控制台输入的参数是a 15s,那么我们要15*1000才能得到时间戳的毫秒时间
                return Tuple2.of(s[0], Long.parseLong(s[1]) * 1000L);
            }
        });

        //设置水位线
        SingleOutputStreamOperator<Tuple2<String, Long>> watermarks2 = initData2.assignTimestampsAndWatermarks(
                // 针对乱序流插入水位线，延迟时间设置为 2s
                WatermarkStrategy.<Tuple2<String, Long>>forBoundedOutOfOrderness(Duration.ofSeconds(0))
                        .withTimestampAssigner(new SerializableTimestampAssigner<Tuple2<String, Long>>() {
                            @Override
                            public long extractTimestamp(Tuple2<String, Long> element, long recordTimestamp) {
                                //指定事件时间
                                return element.f1;
                            }
                        })
        );

        //这里使用intervaljoin
        SingleOutputStreamOperator<Tuple2<String, Long>> resultProcess = watermarks.keyBy(data -> data.f0)
                .intervalJoin(watermarks2.keyBy(data -> data.f0))
                .between(Time.seconds(-5), Time.seconds(5))
                .lowerBoundExclusive()
                .process(new ProcessJoinFunction<Tuple2<String, Long>, Tuple2<String, Long>, Tuple2<String, Long>>() {
                    @Override
                    public void processElement(Tuple2<String, Long> left, Tuple2<String, Long> right, Context ctx, Collector<Tuple2<String, Long>> out) throws Exception {
                        System.out.println("left: " + left);
                        System.out.println("right: " + right);
                        out.collect(Tuple2.of(left.f0, left.f1 + right.f1));
                    }
                });

        resultProcess.print("result: ");

        env.execute();
    }
}

输入

nc -lk 9999
a 1

输入

nc -lk 9998
a 2

输出

left: (a,1000)
right: (a,2000)
result: > (a,3000)

FlinkSql

join内连接

<scala.binary.version>2.12</scala.binary.version>
<dependency>
    <groupId>org.apache.flink</groupId>
    <artifactId>flink-table-planner-blink_${scala.version}</artifactId>
    <version>${flink.version}</version>
</dependency>

测试代码

public class DoubleJoin {
    public static void main(String[] args) throws Exception {
//得到执行环境
        StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();


        DataStreamSource<String> socketTextStream = env.socketTextStream("master", 9999);

        //并行度设置为1才能看到效果，因为如果不为1，那么有些分区的水位线就是负无穷
        //由于自己的水位线是分区里面最小的水位线，那么自己的一直都是负无穷
        //就触发不了水位线的上升
        env.setParallelism(1);

        //第一个参数就一个名字，第二个参数用来表示事件时间
        SingleOutputStreamOperator<Tuple2<String, Long>> initData = socketTextStream.map(new MapFunction<String, Tuple2<String, Long>>() {
            @Override
            public Tuple2<String, Long> map(String value) throws Exception {
                String[] s = value.split(" ");
                //假设我们在控制台输入的参数是a 15s,那么我们要15*1000才能得到时间戳的毫秒时间
                return Tuple2.of(s[0], Long.parseLong(s[1]) * 1000L);
            }
        });


        DataStreamSource<String> socketTextStream2 = env.socketTextStream("master", 9998);

        //并行度设置为1才能看到效果，因为如果不为1，那么有些分区的水位线就是负无穷
        //由于自己的水位线是分区里面最小的水位线，那么自己的一直都是负无穷
        //就触发不了水位线的上升
        env.setParallelism(1);

        //第一个参数就一个名字，第二个参数用来表示事件时间
        SingleOutputStreamOperator<Tuple2<String, Long>> initData2 = socketTextStream2.map(new MapFunction<String, Tuple2<String, Long>>() {
            @Override
            public Tuple2<String, Long> map(String value) throws Exception {
                String[] s = value.split(" ");
                //假设我们在控制台输入的参数是a 15s,那么我们要15*1000才能得到时间戳的毫秒时间
                return Tuple2.of(s[0], Long.parseLong(s[1]) * 1000L);
            }
        });


        //使用FlinkSql
        StreamTableEnvironment tableEnv = StreamTableEnvironment.create(env);
        tableEnv.createTemporaryView("t1", initData);
        tableEnv.createTemporaryView("t2", initData2);

        tableEnv.sqlQuery("select * from t1 join t2 on t1.f0=t2.f0")
                .execute()
                .print();
    }
}

设置数据的过期时间如果不设置默认是一次保存

tableEnv.getConfig().setIdleStateRetention(Duration.ofSeconds(10));

 输入

nc -lk 9999
a 1
a 1
a 1
a 2

输入

nc -lk 9998
a 1
a 1
a 1
a 2

结果

| +I |                              a |                 1000 |                              a |                 2000 |
| +I |                              a |                 1000 |                              a |                 2000 |
| +I |                              a |                 1000 |                              a |                 2000 |
| +I |                              a |                 2000 |                              a |                 2000 |

left join

tableEnv.getConfig().setIdleStateRetention(Duration.ofSeconds(10));

        tableEnv.sqlQuery("select * from t1 left join t2 on t1.f0=t2.f0")
                .execute()
                .print();

说明

使用左外连接的时候，哪个左边就是主表，如果右边有数据和主编进行关联以后，那么主表的数据存活时间10秒会一直更新，就是每读一次就更新一次

right join

tableEnv.getConfig().setIdleStateRetention(Duration.ofSeconds(10));

        tableEnv.sqlQuery("select * from t1 right join t2 on t1.f0=t2.f0")
                .execute()
                .print();

说明

使用右外连接的时候，就是如果左边关联右边的数据进行更新了，那么就是右边的数据的时间就会一直更新

full join

Lookup Join (Mysql)

实验

下面的程序每一条数据过来以后都会去mysql里面进行关联

 pojo

@Data
@AllArgsConstructor
@NoArgsConstructor
public class Event {
    private String id;
    private String base_dic_id;
}

实验程序

public class LookUpTest {
    public static void main(String[] args) throws Exception {
        StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
        DataStreamSource<String> socketTextStream = env.socketTextStream("master", 9999);

        //并行度设置为1才能看到效果，因为如果不为1，那么有些分区的水位线就是负无穷
        //由于自己的水位线是分区里面最小的水位线，那么自己的一直都是负无穷
        //就触发不了水位线的上升
        env.setParallelism(1);

        //第一个参数就一个名字，第二个参数用来表示事件时间
        SingleOutputStreamOperator<Event> sockTest = socketTextStream.map(new MapFunction<String, Event>() {
            @Override
            public Event map(String value) throws Exception {
                String[] initData = value.split(" ");
                String dataone = initData[0];
                String datatow = initData[1];
                return new Event(dataone, datatow);
            }
        });

        //使用FlinkSQL
        StreamTableEnvironment tableEnvironment = StreamTableEnvironment.create(env);
        //从流中得到表结构数据
        Table dataStream = tableEnvironment.fromDataStream(sockTest,
                //下面的数据要和对应的实体类字段对应起来
                $("id"),
                $("base_dic_id"),
                //使用Lookupjoin要加上一个处理时间
                $("pt").proctime());

        //从table创建一个主表
        tableEnvironment.createTemporaryView("t1",dataStream);
//        tableEnvironment.sqlQuery("select * from t1")
//                        .execute().print();

        //使用lookup得到维度退化表
        TableResult tableResult = tableEnvironment.executeSql(
                //这里下面的字段要和数据库里面的字段位置对应起来
                "CREATE TEMPORARY TABLE base_dic( " +
                        "  dic_code STRING, " +
                        "  dic_name STRING " +
                        ") WITH ( " +
                        "  'connector' = 'jdbc', " +
                        "  'username' = 'root', " +
                        "  'password' = 'root', " +
                        //加上缓存但是会牺牲准确性
                        "  'lookup.cache.max-rows' = '10'," +
                        "  'lookup.cache.ttl' = '1 hour'," +
                        "  'url' = 'jdbc:mysql://master:3306/gmall', " +
                        "  'table-name' = 'base_dic' " +
                        ")"
        );

        tableEnvironment.sqlQuery("SELECT t1.id, t1.base_dic_id,t2.dic_code,t2.dic_name  " +
                        "FROM t1  " +
                //要加上FOR SYSTEM_TIME AS OF t1.pt(这个就是上面的处理时间$("pt").proctime())
                        "JOIN base_dic FOR SYSTEM_TIME AS OF t1.pt AS t2  " +
                        "ON t1.id = t2.dic_code")
                .execute()
                .print();
    }
}

数据库里面表的信息为

 控制台输入

nc -lk 9999
10 fkadjsf
10 fhkd

输出的结果为

+----+--------------------------------+--------------------------------+--------------------------------+--------------------------------+
| op |                             id |                    base_dic_id |                       dic_code |                       dic_name |
+----+--------------------------------+--------------------------------+--------------------------------+--------------------------------+
| +I |                             10 |                           fhkd |                             10 |                       单据状态 |

辅助工具类

public class MysqlUtil {
public static String getBaseDicLookUpDDL() {

        return "create table `base_dic`(\n" +
                "`dic_code` string,\n" +
                "`dic_name` string,\n" +
                "`parent_code` string,\n" +
                "`create_time` timestamp,\n" +
                "`operate_time` timestamp,\n" +
                "primary key(`dic_code`) not enforced\n" +
                ")" + MysqlUtil.mysqlLookUpTableDDL("base_dic");
    }

public static String mysqlLookUpTableDDL(String tableName) {

        String ddl = "WITH (\n" +
                "'connector' = 'jdbc',\n" +
                "'url' = 'jdbc:mysql://hadoop102:3306/gmall',\n" +
                "'table-name' = '" + tableName + "',\n" +
                "'lookup.cache.max-rows' = '10',\n" +
                "'lookup.cache.ttl' = '1 hour',\n" +
                "'username' = 'root',\n" +
                "'password' = '000000',\n" +
                "'driver' = 'com.mysql.cj.jdbc.Driver'\n" +
                ")";
        return ddl;
    }
}

Lookup Join(kafka)

        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-json</artifactId>
            <version>${flink.version}</version>
        </dependency>

从maxwell到kafka的topic_db的json格式的数据为

{
    "database": "gmall",
    "table": "base_dic",
    "type": "update",
    "ts": 1658998497,
    "xid": 1602,
    "commit": true,
    "data": {
        "dic_code": "1006",
        "dic_name": "退款完成",
        "parent_code": "10",
        "create_time": null,
        "operate_time": null
    },
    "old": {
        "dic_code": "10066"
    }
}

实时程序

public class KafkaUtil {
    private final static String BOOTSTRAP_SERVERS="master:9092";

    /**
     * Kafka-Source DDL 语句
     *
     * @param topic   数据源主题
     * @param groupId 消费者组
     * @return 拼接好的 Kafka 数据源 DDL 语句
     */
    public static String getKafkaDDL(String topic, String groupId) {

        return " with ('connector' = 'kafka', " +
                " 'topic' = '" + topic + "'," +
                " 'properties.bootstrap.servers' = '" + BOOTSTRAP_SERVERS + "', " +
                " 'properties.group.id' = '" + groupId + "', " +
                " 'format' = 'json', " +
                " 'scan.startup.mode' = 'group-offsets')";
    }

    /**
     * Kafka-Sink DDL 语句
     *
     * @param topic 输出到 Kafka 的目标主题
     * @return 拼接好的 Kafka-Sink DDL 语句
     */
    public static String getUpsertKafkaDDL(String topic) {

        return "WITH ( " +
                "  'connector' = 'upsert-kafka', " +
                "  'topic' = '" + topic + "', " +
                "  'properties.bootstrap.servers' = '" + BOOTSTRAP_SERVERS + "', " +
                "  'key.format' = 'json', " +
                "  'value.format' = 'json' " +
                ")";
    }
}

实现

public class KafkaLookup {
    public static void main(String[] args) throws Exception {
        // TODO 1. 环境准备
        StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
        env.setParallelism(4);
        StreamTableEnvironment tableEnv = StreamTableEnvironment.create(env);

        // 设定 Table 中的时区为本地时区
        tableEnv.getConfig().setLocalTimeZone(ZoneId.of("GMT+8"));

//        // TODO 2. 状态后端设置
//        env.enableCheckpointing(3000L, CheckpointingMode.EXACTLY_ONCE);
//        env.getCheckpointConfig().setMinPauseBetweenCheckpoints(3000L);
//        env.getCheckpointConfig().setCheckpointTimeout(60 * 1000L);
//        env.getCheckpointConfig().enableExternalizedCheckpoints(
//                CheckpointConfig.ExternalizedCheckpointCleanup.RETAIN_ON_CANCELLATION
//        );
//        env.setRestartStrategy(RestartStrategies.failureRateRestart(
//                3, Time.days(1), Time.minutes(1)
//        ));
//        env.setStateBackend(new HashMapStateBackend());
//        env.getCheckpointConfig().setCheckpointStorage("hdfs://hadoop102:8020/ck");
//        System.setProperty("HADOOP_USER_NAME", "atguigu");


        // TODO 3. 从 Kafka 读取业务数据，封装为 Flink SQL 表
        tableEnv.executeSql("" +
                "create table topic_db(   " +
                "`database` string,   " +
                "`table` string,   " +
                "`type` string,   " +
                "`data` map<string, string>,   " +
                "`old` map<string, string>,   " +
                "`ts` string,   " +
                "`proc_time` as PROCTIME()   " +
                //消费topic_db数据
                ")" + KafkaUtil.getKafkaDDL("topic_db", "dwd_trade_cart_add"));

        //打印测试
        tableEnv.sqlQuery("select * from topic_db")
                .execute()
                .print();

        env.execute();

    }
}

输出的结果为

+----+--------------------------------+--------------------------------+--------------------------------+--------------------------------+--------------------------------+--------------------------------+-------------------------+
| op |                       database |                          table |                           type |                           data |                            old |                             ts |               proc_time |
+----+--------------------------------+--------------------------------+--------------------------------+--------------------------------+--------------------------------+--------------------------------+-------------------------+
| +I |                          gmall |                       base_dic |                         update | {parent_code=10, dic_code=1... |               {dic_code=10066} |                     1658998497 | 2022-07-28 16:54:58.077 |

交易域加购维度Lookup join维度降维处理

工具类

public class KafkaUtil {
    private final static String BOOTSTRAP_SERVERS="master:9092";

    /**
     * Kafka-Source DDL 语句
     *
     * @param topic   数据源主题
     * @param groupId 消费者组
     * @return 拼接好的 Kafka 数据源 DDL 语句
     */
    public static String getKafkaDDL(String topic, String groupId) {

        return " with ('connector' = 'kafka', " +
                " 'topic' = '" + topic + "'," +
                " 'properties.bootstrap.servers' = '" + BOOTSTRAP_SERVERS + "', " +
                " 'properties.group.id' = '" + groupId + "', " +
                " 'format' = 'json', " +
                " 'scan.startup.mode' = 'group-offsets')";
    }

    /**
     * Kafka-Sink DDL 语句
     *
     * @param topic 输出到 Kafka 的目标主题
     * @return 拼接好的 Kafka-Sink DDL 语句
     */
    public static String getUpsertKafkaDDL(String topic) {

        return "WITH ( " +
                "  'connector' = 'upsert-kafka', " +
                "  'topic' = '" + topic + "', " +
                "  'properties.bootstrap.servers' = '" + BOOTSTRAP_SERVERS + "', " +
                "  'key.format' = 'json', " +
                "  'value.format' = 'json' " +
                ")";
    }
}
public class MysqlUtils {
    public static String getBaseDicLookUpDDL() {

        return "create table `base_dic`( " +
                "`dic_code` string, " +
                "`dic_name` string, " +
                "`parent_code` string, " +
                "`create_time` timestamp, " +
                "`operate_time` timestamp, " +
                "primary key(`dic_code`) not enforced " +
                ")" + MysqlUtils.mysqlLookUpTableDDL("base_dic");
    }

    public static String mysqlLookUpTableDDL(String tableName) {

        String ddl = "WITH ( " +
                "'connector' = 'jdbc', " +
                "'url' = 'jdbc:mysql://hadoop102:3306/gmall', " +
                "'table-name' = '" + tableName + "', " +
                "'lookup.cache.max-rows' = '10', " +
                "'lookup.cache.ttl' = '1 hour', " +
                "'username' = 'root', " +
                "'password' = '000000', " +
                "'driver' = 'com.mysql.cj.jdbc.Driver' " +
                ")";
        return ddl;
    }
}

应用实现 

public class DwdTradeCartAdd {
    public static void main(String[] args) throws Exception {

        // TODO 1. 环境准备
        StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
        env.setParallelism(4);
        StreamTableEnvironment tableEnv = StreamTableEnvironment.create(env);

        // 设定 Table 中的时区为本地时区
        tableEnv.getConfig().setLocalTimeZone(ZoneId.of("GMT+8"));

        // TODO 2. 状态后端设置
        env.enableCheckpointing(3000L, CheckpointingMode.EXACTLY_ONCE);
        env.getCheckpointConfig().setMinPauseBetweenCheckpoints(3000L);
        env.getCheckpointConfig().setCheckpointTimeout(60 * 1000L);
        env.getCheckpointConfig().enableExternalizedCheckpoints(
                CheckpointConfig.ExternalizedCheckpointCleanup.RETAIN_ON_CANCELLATION
        );
        env.setRestartStrategy(RestartStrategies.failureRateRestart(
                3, Time.days(1), Time.minutes(1)
        ));
        env.setStateBackend(new HashMapStateBackend());
        env.getCheckpointConfig().setCheckpointStorage("hdfs://hadoop102:8020/ck");
        System.setProperty("HADOOP_USER_NAME", "atguigu");

        // TODO 3. 从 Kafka 读取业务数据，封装为 Flink SQL 表
        tableEnv.executeSql("" +
                "create table topic_db( " +
                "`database` string, " +
                "`table` string, " +
                "`type` string, " +
                "`data` map<string, string>, " +
                "`old` map<string, string>, " +
                "`ts` string, " +
                "`proc_time` as PROCTIME() " +
                ")" + KafkaUtil.getKafkaDDL("topic_db", "dwd_trade_cart_add"));

        // TODO 4. 读取购物车表数据
        Table cartAdd = tableEnv.sqlQuery("" +
                "select " +
                "data['id'] id, " +
                "data['user_id'] user_id, " +
                "data['sku_id'] sku_id, " +
                "data['source_id'] source_id, " +
                "data['source_type'] source_type, " +
                "if(`type` = 'insert', " +
                "data['sku_num'],cast((cast(data['sku_num'] as int) - cast(`old`['sku_num'] as int)) as string)) sku_num, " +
                "ts, " +
                "proc_time " +
                "from `topic_db`  " +
                "where `table` = 'cart_info' " +
                "and (`type` = 'insert' " +
                "or (`type` = 'update'  " +
                "and `old`['sku_num'] is not null  " +
                "and cast(data['sku_num'] as int) > cast(`old`['sku_num'] as int)))");
        tableEnv.createTemporaryView("cart_add", cartAdd);

        // TODO 5. 建立 MySQL-LookUp 字典表
        tableEnv.executeSql(MysqlUtils.getBaseDicLookUpDDL());

        // TODO 6. 关联两张表获得加购明细表
        Table resultTable = tableEnv.sqlQuery("select " +
                "cadd.id, " +
                "user_id, " +
                "sku_id, " +
                "source_id, " +
                "source_type, " +
                "dic_name source_type_name, " +
                "sku_num, " +
                "ts " +
                "from cart_add cadd " +
                "left join base_dic for system_time as of cadd.proc_time as dic " +
                "on cadd.source_type=dic.dic_code");
        tableEnv.createTemporaryView("result_table", resultTable);

        // TODO 7. 建立 Upsert-Kafka dwd_trade_cart_add 表
        tableEnv.executeSql("" +
                "create table dwd_trade_cart_add( " +
                "id string, " +
                "user_id string, " +
                "sku_id string, " +
                "source_id string, " +
                "source_type_code string, " +
                "source_type_name string, " +
                "sku_num string, " +
                "ts string, " +
                "primary key(id) not enforced " +
                ")" + KafkaUtil.getUpsertKafkaDDL("dwd_trade_cart_add"));

        // TODO 8. 将关联结果写入 Upsert-Kafka 表
        tableEnv.executeSql("" +
                "insert into dwd_trade_cart_add select * from result_table");
    }
}

pom.xml

<build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <configuration>
                    <source>8</source>
                    <target>8</target>
                </configuration>
            </plugin>

            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-assembly-plugin</artifactId>
                <version>3.0.0</version>
                <configuration>
                    <descriptorRefs>
                        <descriptorRef>jar-with-dependencies</descriptorRef>
                    </descriptorRefs>
                </configuration>
                <executions>
                    <execution>
                        <id>make-assembly</id>
                        <phase>package</phase>
                        <goals>
                            <goal>single</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>

    <properties>
        <flink.version>1.13.0</flink.version>
        <java.version>1.8</java.version>
        <scala.binary.version>2.12</scala.binary.version>
        <slf4j.version>1.7.30</slf4j.version>
        <scala.version>2.12</scala.version>
    </properties>
    <dependencies>
        <!-- 引入 Flink 相关依赖-->
        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-java</artifactId>
            <version>${flink.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-streaming-java_${scala.binary.version}</artifactId>
            <version>${flink.version}</version>
        </dependency>

        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-cep_${scala.binary.version}</artifactId>
            <version>${flink.version}</version>
        </dependency>

        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-connector-kafka_${scala.binary.version}</artifactId>
            <version>${flink.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-clients_${scala.binary.version}</artifactId>
            <version>${flink.version}</version>
        </dependency>
        <!-- 引入日志管理相关依赖-->
        <dependency>
            <groupId>org.slf4j</groupId>
            <artifactId>slf4j-api</artifactId>
            <version>${slf4j.version}</version>
        </dependency>
        <dependency>
            <groupId>org.slf4j</groupId>
            <artifactId>slf4j-log4j12</artifactId>
            <version>${slf4j.version}</version>
        </dependency>

        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-table-api-java-bridge_${scala.binary.version}</artifactId>
            <version>${flink.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-table-planner-blink_${scala.binary.version}</artifactId>
            <version>${flink.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.logging.log4j</groupId>
            <artifactId>log4j-to-slf4j</artifactId>
            <version>2.14.0</version>
        </dependency>

        <dependency>
            <groupId>org.apache.bahir</groupId>
            <artifactId>flink-connector-redis_2.11</artifactId>
            <version>1.0</version>
        </dependency>

        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-connector-elasticsearch7_${scala.binary.version}</artifactId>
            <version>${flink.version}</version>
        </dependency>


        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-connector-jdbc_${scala.binary.version}</artifactId>
            <version>${flink.version}</version>
        </dependency>
        <dependency>
            <groupId>mysql</groupId>
            <artifactId>mysql-connector-java</artifactId>
            <version>5.1.47</version>
        </dependency>


        <!--        这里的依赖是一个 Java 的“桥接器”（bridge），主要就是负责 Table API 和下层 DataStream-->
        <!--        API 的连接支持，按照不同的语言分为 Java 版和 Scala 版。-->
        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-table-api-java-bridge_${scala.binary.version}</artifactId>
            <version>${flink.version}</version>
        </dependency>

        <!--        如果我们希望在本地的集成开发环境（IDE）里运行 Table API 和 SQL，还需要引入以下-->
        <!--        依赖-->
        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-table-planner-blink_${scala.binary.version}</artifactId>
            <version>${flink.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-streaming-scala_${scala.binary.version}</artifactId>
            <version>${flink.version}</version>
        </dependency>


        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-cep_${scala.binary.version}</artifactId>
            <version>${flink.version}</version>
        </dependency>


        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-table-planner-blink_${scala.version}</artifactId>
            <version>${flink.version}</version>
        </dependency>
        <dependency>
            <groupId>org.projectlombok</groupId>
            <artifactId>lombok</artifactId>
            <version>RELEASE</version>
            <scope>compile</scope>
        </dependency>


        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-table-planner-blink_${scala.version}</artifactId>
            <version>${flink.version}</version>
        </dependency>

        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-json</artifactId>
            <version>${flink.version}</version>
        </dependency>

    </dependencies>

图解逻辑

 

工作变成艺术
关注
  • 1
    点赞
  • 踩
  • 1
    收藏
    觉得还不错? 一键收藏
  • 打赏
    打赏
  • 0
    评论
专栏目录
Flink-java案例：在一小时内每隔5s统计一次商品pv的TopN
m0_48379126的博客
12-29 1600
package com.zyj.day04; import org.apache.flink.api.common.eventtime.SerializableTimestampAssigner; import org.apache.flink.api.common.eventtime.WatermarkStrategy; import org.apache.flink.api.common.functions.AggregateFunction; import org.apache.flink.api.
cacheable-lookup:尊重TTL的可缓存dns.lookup（…）
05-10
可缓存查找 尊重TTL的可缓存 :party_popper: 发出大量HTTP请求？ 您可以通过缓存DNS查找来节省一些时间 :high_voltage: 用法 使用lookup选项 const http = require ( 'http' ) ; const CacheableLookup = require ( 'cacheable-lookup' ) ; const cacheable = new CacheableLookup ( ) ; http . get ( 'http://example.com' , { lookup : cacheable . lookup } , response => { // Handle the response here } ) ; 将CacheableLookup附加到代理 const http = require ( 'http' ) ; const CacheableLookup
（5）3.4DWD层5
weixin_42871374的博客
04-09 330
3.4DWD层 对ODS层数据进行判空过滤。对商品分类表进行维度退化（降维）。 3.4.1 创建订单表 hive (gmall)> drop table if exists dwd_order_info; create external table dwd_order_info ( id string COMMENT ‘’, total_amount decimal(10,2) COMMEN...
实时数仓-涉及维度退化的事实表
芜湖
06-07 588
[关键字]:Flink SQL、Flink 双流Join、Flink 动态表与持续查询、宽表、维度退化
flink1.14 sql基础语法（一） flink sql表查询详解
qq_44665283的博客
07-21 7140
flink sql表查询详解
FLINK-SQL
JinVijay的博客
03-01 1841
关联MySQL中的维度表示例 /** * @Description: * 从Kafka中读数据，然后关联MySQL的维表，最后将结果输出到MySQL中 */ public class MySQLDimensionDemo { public static void main(String[] args) { StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment
flink-table-planner-2.12-1.14.3-API文档-中文版.zip
04-07
赠送jar包：flink-table-planner_2.12-1.14.3.jar 赠送原API文档：flink-table-planner_2.12-1.14.3-javadoc.jar 赠送源代码：flink-table-planner_2.12-1.14.3-sources.jar 包含翻译后的API文档：flink-table-...
flink-table-runtime-blink_2.11-1.10.0-API文档-中文版.zip
04-23
赠送jar包：flink-table-runtime-blink_2.11-1.10.0.jar； 赠送原API文档：flink-table-runtime-blink_2.11-1.10.0-javadoc.jar； 赠送源代码：flink-table-runtime-blink_2.11-1.10.0-sources.jar； 赠送Maven依赖...
flink-table-planner-blink_2.11-1.10.0-API文档-中文版.zip
04-23
赠送jar包：flink-table-planner-blink_2.11-1.10.0.jar； 赠送原API文档：flink-table-planner-blink_2.11-1.10.0-javadoc.jar； 赠送源代码：flink-table-planner-blink_2.11-1.10.0-sources.jar； 赠送Maven依赖...
apache-doris-flink-connector-1.14_2.12-1.0.3
05-05
Flink Doris Connector（apache-doris-flink-connector-1.14_2.12-1.0.3-incubating-src.tar.gz） Flink Doris Connector Version：1.0.3 Flink Version：1.14 Scala Version：2.12 Apache Doris是一个现代MPP...
flink-1.14.0-bin-scala_2.11.tgz
07-18
flink-1.14.0-bin-scala_2.11.tgz
Flink SQL空闲状态保留时间（idle state retention time）实现原理
LittleMagic's Blog
01-06 5154
前言 如果要列举Flink SQL新手有可能犯的错误，笔者认为其中之一就是忘记设置空闲状态保留时间导致状态爆炸。2021年的第一篇技术文，时间很紧张，聊聊这个简单的话题吧。 为什么要设置 如果我们在数据流上进行分组查询，分组处理产生的结果(不仅仅是聚合结果)会作为中间状态存储下来。随着分组key的不断增加，状态自然也会不断膨胀。但是这些状态数据基本都有时效性，不必永久保留。例如，使用Top-N语法...
以超卖案例深入理解Redis分布式锁
黑马程序员官方博客
11-10 1336
本文介绍了超卖问题产生的原因：操作不具备原子性，同时提出了集中解决思路。synchronized锁，无法保证多实例下的线程安全setnx手动实现，坑很多、代码较为复杂redisson实现，能够保证多实例下线程安全，代码简单可靠。
Flink SQL join类型
chanyue123的博客
03-07 1637
flink sql join类型讲解
数据仓库和建模
denganming1214的博客
07-04 2178
代理键是由数据仓库处理过程中产生的、与业务本身无关的、唯一标识维度表中一条记录并充当维度表主键的列，也是描述维度表与事实表关系的纽带。在现实世界中，每一个操作型事件，基本都是发生在实体之间的，伴随着这种操作事件的发生，会产生可度量的值，而这个过程就产生了一个事实表，存储了每一个可度量的事件。订单表：（维度表主键）时间外建，用户外建，商品外键，（度量数据）下单数量、价格、（事件描述信息）状态、付款时间。事实表为订单表、子订单表，维度包括商品维度、用户维度、商家维度、区域维度、时间维度。
Flink 优化(六) --------- FlinkSQL 调优
在森林里麋了鹿
04-12 1529
Flink SQL 调优
Flink实时仓库-DWD层(left-join后写入kafka)模板代码
S1124654的博客
07-28 470
Flink实时仓库-DWD层(left-join后写入kafka)模板代码
flink sql中指定时间字段
热门推荐
litlit023的专栏
01-22 1万+
1. 通过proctime AS PROCTIME() 来指定 如下面flink ddl中使用 proctime AS PROCTIME() 来生产一个新的计算列，该计算列可当作处理时间来对待。 计算列常常被用在定义时间属性(见另一篇文章Flink Table API&SQL编程指南之时间属性(3)，可以通过PROCTIME()函数定义处理时间属性，语法为proc AS PROCTIME()。 除此之外，计算列可以被用作提取事件时间列，因为原始的事件时间可能不是TIMESTAMP(3)类型或者
Flink SQL 优化实战 - 维表 JOIN 优化
cloudbigdata的博客
05-27 6372
从源码层面分析维表 Join 原理及调优方案
flink ods层到dwd层
最新发布
04-28
而DWD层则是数据仓库明细层，主要用于存储处理后的数据，保证数据的完整性、准确性和一致性，并将其按照业务维度进行分类、聚合和汇总，形成可供分析的数据集合。同时，DWD层还需要对数据进行质量检测和异常处理，以...

“相关推荐”对你有帮助么？

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包

打赏作者

工作变成艺术

你的鼓励将是我创作的最大动力

¥1 ¥2 ¥4 ¥6 ¥10 ¥20
扫码支付：¥1
获取中
扫码支付

您的余额不足，请更换扫码支付或充值

打赏作者

实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值