Flink Icerberg 离线数仓-项目准备（三）

已于 2022-07-04 15:57:56 修改
阅读量737 收藏
点赞数 1
分类专栏： Apache 文章标签： flink 大数据 数据库 数据仓库
于 2022-07-04 15:42:54 首次发布
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/wank1259162/article/details/125600993
版权

本节主要是创建模拟数据，包括事实表和维度表，并通过Flink映射为Icerberg数据表，构成DWD层数据。

测试数据目录

ODS数据准备

Base_ad_log.log 数据格式

{"adid":"0","adname":"注册弹窗广告0","dn":"webA"}
{"adid":"1","adname":"注册弹窗广告1","dn":"webA"}
{"adid":"2","adname":"注册弹窗广告2","dn":"webA"}
{"adid":"3","adname":"注册弹窗广告3","dn":"webA"}
{"adid":"4","adname":"注册弹窗广告4","dn":"webA"}
{"adid":"5","adname":"注册弹窗广告5","dn":"webA"}
{"adid":"6","adname":"注册弹窗广告6","dn":"webA"}
{"adid":"7","adname":"注册弹窗广告7","dn":"webA"}
{"adid":"8","adname":"注册弹窗广告8","dn":"webA"}
{"adid":"9","adname":"注册弹窗广告9","dn":"webA"}

Base_website.log  数据格式

{"createtime":"2000-01-01","creator":"admin","delete":"0","dn":"webA","siteid":"0","sitename":"百度","siteurl":"wwww.baidu.com/webA"}
{"createtime":"2000-01-01","creator":"admin","delete":"0","dn":"webA","siteid":"1","sitename":"163","siteurl":"www.163.com/webA"}
{"createtime":"2000-01-01","creator":"admin","delete":"0","dn":"webA","siteid":"2","sitename":"114","siteurl":"www.114.com/webA"}
{"createtime":"2000-01-01","creator":"admin","delete":"0","dn":"webA","siteid":"3","sitename":"126","siteurl":"www.126.com/webA"}
{"createtime":"2000-01-01","creator":"admin","delete":"0","dn":"webA","siteid":"4","sitename":"谷歌","siteurl":"www.google.com/webA"}
{"createtime":"2000-01-01","creator":"admin","delete":"0","dn":"webB","siteid":"0","sitename":"百度","siteurl":"wwww.baidu.com/webB"}

Member.log 数据格式

{"ad_id":"9","birthday":"1997-11-16","dn":"webA","dt":"20190722","email":"test@126.com","fullname":"王0","iconurl":"-","lastlogin":"-","mailaddr":"-","memberlevel":"1","password":"123456","paymoney":"-","phone":"13711235451","qq":"10000","register":"2015-04-05","regupdatetime":"-","uid":"0","unitname":"-","userip":"222.42.116.199","zipcode":"-"}
{"ad_id":"5","birthday":"1997-03-08","dn":"webA","dt":"20190722","email":"test@126.com","fullname":"王1","iconurl":"-","lastlogin":"-","mailaddr":"-","memberlevel":"4","password":"123456","paymoney":"-","phone":"13711235451","qq":"10000","register":"2017-10-13","regupdatetime":"-","uid":"1","unitname":"-","userip":"106.92.133.13","zipcode":"-"}
{"ad_id":"1","birthday":"1998-10-18","dn":"webA","dt":"20190722","email":"test@126.com","fullname":"王2","iconurl":"-","lastlogin":"-","mailaddr":"-","memberlevel":"7","password":"123456","paymoney":"-","phone":"13711235451","qq":"10000","register":"2018-03-16","regupdatetime":"-","uid":"2","unitname":"-","userip":"139.200.218.184","zipcode":"-"}
{"ad_id":"2","birthday":"1970-10-27","dn":"webA","dt":"20190722","email":"test@126.com","fullname":"王3","iconurl":"-","lastlogin":"-","mailaddr":"-","memberlevel":"2","password":"123456","paymoney":"-","phone":"13711235451","qq":"10000","register":"2016-08-22","regupdatetime":"-","uid":"3","unitname":"-","userip":"121.77.205.103","zipcode":"-"}
{"ad_id":"3","birthday":"1975-06-16","dn":"webA","dt":"20190722","email":"test@126.com","fullname":"王4","iconurl":"-","lastlogin":"-","mailaddr":"-","memberlevel":"8","password":"123456","paymoney":"-","phone":"13711235451","qq":"10000","register":"2015-08-14","regupdatetime":"-","uid":"4","unitname":"-","userip":"121.77.66.4","zipcode":"-"}

Member_Regtype.log  数据格式

{"appkey":"-","appregurl":"http:www.webA.com/sale/register/index.html","bdp_uuid":"-","createtime":"2017-03-30","dn":"webA","domain":"-","dt":"20190722","isranreg":"-","regsource":"0","uid":"0","websiteid":"4"}
{"appkey":"-","appregurl":"http:www.webA.com/sale/register/index.html","bdp_uuid":"-","createtime":"2018-02-20","dn":"webA","domain":"-","dt":"20190722","isranreg":"-","regsource":"3","uid":"1","websiteid":"2"}
{"appkey":"-","appregurl":"http:www.webA.com/product10/register/aa/index.html","bdp_uuid":"-","createtime":"2017-09-18","dn":"webA","domain":"-","dt":"20190722","isranreg":"-","regsource":"1","uid":"2","websiteid":"1"}
{"appkey":"-","appregurl":"http:www.webA.com/hhh/wwww/index.html","bdp_uuid":"-","createtime":"2017-10-18","dn":"webA","domain":"-","dt":"20190722","isranreg":"-","regsource":"4","uid":"3","websiteid":"2"}
{"appkey":"-","appregurl":"http:www.webA.com/product/register/index.html","bdp_uuid":"-","createtime":"2017-01-20","dn":"webA","domain":"-","dt":"20190722","isranreg":"-","regsource":"2","uid":"4","websiteid":"3"}
{"appkey":"-","appregurl":"http:www.webA.com/sale/register/index.html","bdp_uuid":"-","createtime":"2019-01-01","dn":"webA","domain":"-","dt":"20190722","isranreg":"-","regsource":"4","uid":"5","websiteid":"2"}

Pcenter_Mem_Paymoney.log 数据格式

{"dn":"webA","dt":"20190722","paymoney":"143.25","siteid":"0","uid":"85853","vip_id":"0"}
{"dn":"webA","dt":"20190722","paymoney":"299.71","siteid":"1","uid":"71868","vip_id":"3"}
{"dn":"webA","dt":"20190722","paymoney":"420.72","siteid":"2","uid":"93104","vip_id":"3"}
{"dn":"webA","dt":"20190722","paymoney":"477.82","siteid":"1","uid":"86239","vip_id":"1"}
{"dn":"webA","dt":"20190722","paymoney":"133.38","siteid":"3","uid":"24540","vip_id":"2"}
{"dn":"webA","dt":"20190722","paymoney":"264.61","siteid":"3","uid":"22827","vip_id":"1"}
{"dn":"webA","dt":"20190722","paymoney":"909.87","siteid":"2","uid":"91425","vip_id":"3"}

Pcenter_Mem_Vip_Level.log 数据格式

{"discountval":"-","dn":"webA","end_time":"2018-12-14","last_modify_time":"2018-12-14","max_free":"-","min_free":"-","next_level":"-","operator":"update","start_time":"2015-10-27","vip_id":"0","vip_level":"普通会员"}
{"discountval":"-","dn":"webA","end_time":"2016-11-25","last_modify_time":"2016-11-25","max_free":"-","min_free":"-","next_level":"-","operator":"update","start_time":"2015-08-18","vip_id":"1","vip_level":"白金"}
{"discountval":"-","dn":"webA","end_time":"2016-03-04","last_modify_time":"2016-03-04","max_free":"-","min_free":"-","next_level":"-","operator":"update","start_time":"2015-04-06","vip_id":"2","vip_level":"银卡"}
{"discountval":"-","dn":"webA","end_time":"2017-12-21","last_modify_time":"2017-12-21","max_free":"-","min_free":"-","next_level":"-","operator":"update","start_time":"2016-03-12","vip_id":"3","vip_level":"金卡"}
{"discountval":"-","dn":"webA","end_time":"2016-04-19","last_modify_time":"2016-04-19","max_free":"-","min_free":"-","next_level":"-","operator":"update","start_time":"2015-04-10","vip_id":"4","vip_level":"钻石"}

项目准备

项目POM文件

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <parent>
        <artifactId>iceberg-demo</artifactId>
        <groupId>com.iceberg</groupId>
        <version>1.0-SNAPSHOT</version>
    </parent>
    <modelVersion>4.0.0</modelVersion>

    <artifactId>icberg-flink-demo</artifactId>
    <properties>
        <flink.version>1.11.0</flink.version>
        <scala.version>2.12.10</scala.version>
        <scala.binary.version>2.12</scala.binary.version>
        <log4j.version>1.2.17</log4j.version>
        <slf4j.version>1.7.22</slf4j.version>
        <iceberg.version>0.11.1</iceberg.version>
    </properties>

    <dependencies>
        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-java</artifactId>
            <version>${flink.version}</version>
            <exclusions>
                <exclusion>
                    <groupId>log4j</groupId>
                    <artifactId>*</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>org.slf4j</groupId>
                    <artifactId>slf4j-log4j12</artifactId>
                </exclusion>
            </exclusions>
        </dependency>
        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-streaming-java_${scala.binary.version}</artifactId>
            <version>${flink.version}</version>
            <exclusions>
                <exclusion>
                    <groupId>log4j</groupId>
                    <artifactId>*</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>org.slf4j</groupId>
                    <artifactId>slf4j-log4j12</artifactId>
                </exclusion>
            </exclusions>
        </dependency>
        <dependency>
            <groupId>org.scala-lang</groupId>
            <artifactId>scala-library</artifactId>
            <version>${scala.version}</version>
        </dependency>

        <!-- https://mvnrepository.com/artifact/org.apache.flink/flink-table -->

        <!-- https://mvnrepository.com/artifact/org.apache.flink/flink-table-common -->
        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-table-common</artifactId>
            <version>${flink.version}</version>
        </dependency>
        <!-- https://mvnrepository.com/artifact/org.apache.flink/flink-table-api-java -->
        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-table-api-java</artifactId>
            <version>${flink.version}</version>
        </dependency>
        <!-- https://mvnrepository.com/artifact/org.apache.flink/flink-table-api-java-bridge -->
        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-table-api-java-bridge_${scala.binary.version}</artifactId>
            <version>${flink.version}</version>
        </dependency>
        <!-- https://mvnrepository.com/artifact/org.apache.flink/flink-table-planner -->
        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-table-planner_${scala.binary.version}</artifactId>
            <version>${flink.version}</version>
        </dependency>

        <!-- https://mvnrepository.com/artifact/org.apache.flink/flink-table-planner-blink -->
        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-table-planner-blink_${scala.binary.version}</artifactId>
            <version>${flink.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.iceberg</groupId>
            <artifactId>iceberg-flink-runtime</artifactId>
            <version>0.11.1</version>
        </dependency>
        <!-- https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-common -->
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-client</artifactId>
            <version>3.1.3</version>
        </dependency>
        <!-- https://mvnrepository.com/artifact/org.apache.flink/flink-clients -->
        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-clients_${scala.binary.version}</artifactId>
            <version>${flink.version}</version>
        </dependency>
        <dependency>
            <groupId>com.alibaba</groupId>
            <artifactId>fastjson</artifactId>
            <version>1.2.76</version>
        </dependency>

        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-connector-kafka-0.10_${scala.binary.version}</artifactId>
            <version>${flink.version}</version>
        </dependency>

    </dependencies>

    <build>
        <plugins>
            <plugin>
                <groupId>org.scala-tools</groupId>
                <artifactId>maven-scala-plugin</artifactId>
                <version>2.15.1</version>
                <executions>
                    <execution>
                        <id>compile-scala</id>
                        <goals>
                            <goal>add-source</goal>
                            <goal>compile</goal>
                        </goals>
                    </execution>
                    <execution>
                        <id>test-compile-scala</id>
                        <goals>
                            <goal>add-source</goal>
                            <goal>testCompile</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-assembly-plugin</artifactId>
                <configuration>
                    <archive>
                        <manifest>
                        </manifest>
                    </archive>
                    <descriptorRefs>
                        <descriptorRef>jar-with-dependencies</descriptorRef>
                    </descriptorRefs>
                </configuration>
            </plugin>
        </plugins>
    </build>
</project>

文件目录结构

 数据导入准备

1. 拷贝测试数据目录下的文件到/tmp/warehouse/ods/下面；
2. 创建测试表com.iceberg.warhouse.meta.CreateTables

创建表CreateTables

1.使用Hadoop Catalog存储表元信息

2.使用本地文件系统，便于测试

package com.iceberg.warhouse.meta;

import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;

/**
 *   构建Icerberg表
 */

public class CreateTables {

    public static void main(String[] args) {

        //获取表环境
        StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
        StreamTableEnvironment tableEnv = StreamTableEnvironment.create(env);

        //tableEnv.executeSql("SET execution.checkpointing.interval = 3s");
        tableEnv.executeSql("CREATE CATALOG hadoop_catalog WITH (\n" +
                "  'type'='iceberg',\n" +
                "  'catalog-type'='hadoop',\n" +
                "  'warehouse'='file:///tmp/warehouse/iceberg',\n" +
                "  'property-version'='1'\n" +
                ")");

        tableEnv.executeSql("USE CATALOG hadoop_catalog");

        //dwd_member
        tableEnv.executeSql("create table iceberg.dwd_member(\n" +
                "   uid int,\n" +
                "   ad_id int,\n" +
                "   birthday string,\n" +
                "   email string,\n" +
                "   fullname string,\n" +
                "   iconurl string,\n" +
                "   lastlogin string,\n" +
                "   mailaddr string,\n" +
                "   memberlevel string,\n" +
                "   password string,\n" +
                "   paymoney string,\n" +
                "   phone string,\n" +
                "   qq string,\n" +
                "   register string,\n" +
                "   regupdatetime string,\n" +
                "   unitname string,\n" +
                "   userip string,\n" +
                "   zipcode string,\n" +
                "   dt string)\n" +
                "   partitioned by(dt)");

        //dwd_member_regtype
        tableEnv.executeSql("create table iceberg.dwd_member_regtype(\n" +
                "  uid int,\n" +
                "  appkey string,\n" +
                "  appregurl string,\n" +
                "  bdp_uuid string,\n" +
                "  createtime timestamp,\n" +
                "  isranreg string,\n" +
                "  regsource string,\n" +
                "  regsourcename string,\n" +
                "  websiteid int,\n" +
                "  dt string)\n" +
                "  partitioned by(dt)");

        //dwd_base_ad
        tableEnv.executeSql("create table iceberg.dwd_base_ad(\n" +
                "adid int,\n" +
                "adname string,\n" +
                "dn string)\n" +
                "partitioned by (dn) ");

        //dwd_base_website
        tableEnv.executeSql(" create table iceberg.dwd_base_website(\n" +
                "  siteid int,\n" +
                "  sitename string,\n" +
                "  siteurl string,\n" +
                " `delete` int,\n" +
                "  createtime timestamp,\n" +
                "  creator string,\n" +
                "  dn string)\n" +
                "partitioned by (dn) ");

        //dwd_pcentermempaymoney
        tableEnv.executeSql("create table iceberg.dwd_pcentermempaymoney(\n" +
                "  uid int,\n" +
                "  paymoney string,\n" +
                "  siteid int,\n" +
                "  vip_id int,\n" +
                "  dt string,\n" +
                "  dn string)\n" +
                " partitioned by(dt,dn)");

        //dwd_vip_level
        tableEnv.executeSql(" create table iceberg.dwd_vip_level(\n" +
                "   vip_id int,\n" +
                "   vip_level string,\n" +
                "   start_time timestamp,\n" +
                "   end_time timestamp,\n" +
                "   last_modify_time timestamp,\n" +
                "   max_free string,\n" +
                "   min_free string,\n" +
                "   next_level string,\n" +
                "   operator string,\n" +
                "   dn string)\n" +
                " partitioned by(dn)");

        //dws_member
        tableEnv.executeSql("create table iceberg.dws_member(\n" +
                "  uid int,\n" +
                "  ad_id int,\n" +
                "  fullname string,\n" +
                "  iconurl string,\n" +
                "  lastlogin string,\n" +
                "  mailaddr string,\n" +
                "  memberlevel string,\n" +
                "  password string,\n" +
                "  paymoney string,\n" +
                "  phone string,\n" +
                "  qq string,\n" +
                "  register string,\n" +
                "  regupdatetime string,\n" +
                "  unitname string,\n" +
                "  userip string,\n" +
                "  zipcode string,\n" +
                "  appkey string,\n" +
                "  appregurl string,\n" +
                "  bdp_uuid string,\n" +
                "  reg_createtime string,\n" +
                "  isranreg string,\n" +
                "  regsource string,\n" +
                "  regsourcename string,\n" +
                "  adname string,\n" +
                "  siteid int,\n" +
                "  sitename string,\n" +
                "  siteurl string,\n" +
                "  site_delete string,\n" +
                "  site_createtime string,\n" +
                "  site_creator string,\n" +
                "  vip_id int,\n" +
                "  vip_level string,\n" +
                "  vip_start_time string,\n" +
                "  vip_end_time string,\n" +
                "  vip_last_modify_time string,\n" +
                "  vip_max_free string,\n" +
                "  vip_min_free string,\n" +
                "  vip_next_level string,\n" +
                "  vip_operator string,\n" +
                " dt string,\n" +
                "dn string)\n" +
                "partitioned by(dt,dn)");

        //ads_register_appregurlnum
        tableEnv.executeSql(" create  table iceberg.ads_register_appregurlnum(\n" +
                "  appregurl string,\n" +
                "  num bigint,\n" +
                "  dt string,\n" +
                "  dn string)\n" +
                "partitioned by(dt)");

        //ads_register_top3memberpay
        tableEnv.executeSql(" create table iceberg.ads_register_top3memberpay(\n" +
                "  uid int,\n" +
                "  memberlevel string,\n" +
                "  register string,\n" +
                "  appregurl string,\n" +
                "  regsourcename string,\n" +
                "  adname string,\n" +
                "  sitename string,\n" +
                "  vip_level string,\n" +
                "  paymoney decimal(10,4),\n" +
                "  rownum BigInt,\n" +
                "dn string,\n" +
                " dt string)\n" +
                "partitioned by(dt,rownum,memberlevel)");
    }

}

将ODS数据抽象成Icerberg表

/**
 * 将ODS数据抽象成IcerBerg表
 */
public class DwdIcebergController {
    public static DwdIcebergSerivce dwdIcebergSerivce;

    public static void main(String[] args) throws Exception {

        StreamExecutionEnvironment env = StreamExecutionEnvironment.createLocalEnvironment();
        dwdIcebergSerivce = new DwdIcebergSerivce();

        // 测试数据存放路径,需要把测试数据下面的数据全部拷贝到/tmp/warehouse/ods/下面；
        String basePath = "file:///tmp/warehouse/ods/";

        //使用Hadoop本地路径，Icerberg数据保存到/tmp/warehouse/iceberg/iceberg/；
        String catalogPath = "file:///tmp/warehouse/iceberg/iceberg/";

        dwdIcebergSerivce.readOdsData(env, basePath, catalogPath);
        env.execute();
    }
}

DwdIcebergSerivce具体实现

此处将模拟数据文件，转为DataStream，然后输出到Icerberg。

package com.iceberg.warhouse.service;

import com.alibaba.fastjson.JSONObject;
import org.apache.flink.streaming.api.datastream.DataStream;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.table.data.GenericRowData;
import org.apache.flink.table.data.RowData;
import org.apache.flink.table.data.StringData;
import org.apache.flink.table.data.TimestampData;
import org.apache.iceberg.flink.TableLoader;
import org.apache.iceberg.flink.sink.FlinkSink;

import java.time.LocalDate;
import java.time.LocalDateTime;
import java.time.format.DateTimeFormatter;

public class DwdIcebergSerivce {

    //basePath= file:///tmp/warehouse/ods/
    //catalogPath=file:///tmp/warehouse/iceberg/iceberg/
    public void readOdsData(StreamExecutionEnvironment env, String basePath, String catalogPath) {

        DataStream<String> baseadDS = env.readTextFile(basePath + "baseadlog.log");
        DataStream<RowData> baseadInput = baseadDS.map(item -> {
            JSONObject jsonObject = JSONObject.parseObject(item);
            GenericRowData rowData = new GenericRowData(3);
            rowData.setField(0, jsonObject.getIntValue("adid"));
            rowData.setField(1, StringData.fromString(jsonObject.getString("adname")));
            rowData.setField(2, StringData.fromString(jsonObject.getString("dn")));
            return rowData;
        });
        TableLoader dwdbaseadTable = TableLoader.fromHadoopTable(catalogPath + "dwd_base_ad");
        FlinkSink.forRowData(baseadInput).tableLoader(dwdbaseadTable).overwrite(true).build();


        DataStream<String> basewebsiteDS = env.readTextFile(basePath + "baswewebsite.log");
        DataStream<RowData> basewebsiteInput = basewebsiteDS.map(item -> {
            JSONObject jsonObject = JSONObject.parseObject(item);
            GenericRowData rowData = new GenericRowData(7);
            rowData.setField(0, jsonObject.getIntValue("siteid"));
            rowData.setField(1, StringData.fromString(jsonObject.getString("sitename")));
            rowData.setField(2, StringData.fromString(jsonObject.getString("siteurl")));
            rowData.setField(3, jsonObject.getIntValue("delete"));
            LocalDateTime localDateTime = LocalDate.parse(jsonObject.getString("createtime"), DateTimeFormatter.ofPattern("yyyy-MM-dd"))
                    .atStartOfDay();
            rowData.setField(4, TimestampData.fromLocalDateTime(localDateTime));
            rowData.setField(5, StringData.fromString(jsonObject.getString("creator")));
            rowData.setField(6, StringData.fromString(jsonObject.getString("dn")));
            return rowData;
        });
        TableLoader dwdbasewebsiteTable = TableLoader.fromHadoopTable(catalogPath + "dwd_base_website");
        FlinkSink.forRowData(basewebsiteInput).tableLoader(dwdbasewebsiteTable).overwrite(true).build();


        DataStream<String> memberDS = env.readTextFile(basePath + "member.log");
        DataStream<RowData> memberInput = memberDS.map(item -> {
            JSONObject jsonObject = JSONObject.parseObject(item);
            GenericRowData rowData = new GenericRowData(19);
            rowData.setField(0, jsonObject.getIntValue("uid"));
            rowData.setField(1, jsonObject.getIntValue("ad_id"));
            rowData.setField(2, StringData.fromString(jsonObject.getString("birthday")));
            rowData.setField(3, StringData.fromString(jsonObject.getString("email")));
            rowData.setField(4, StringData.fromString(jsonObject.getString("fullname")));
            rowData.setField(5, StringData.fromString(jsonObject.getString("iconurl")));
            rowData.setField(6, StringData.fromString(jsonObject.getString("lastlogin")));
            rowData.setField(7, StringData.fromString(jsonObject.getString("mailaddr")));
            rowData.setField(8, StringData.fromString(jsonObject.getString("memberlevel")));
            rowData.setField(9, StringData.fromString(jsonObject.getString("password")));
            rowData.setField(10, StringData.fromString(jsonObject.getString("paymoney")));
            rowData.setField(11, StringData.fromString(jsonObject.getString("phone")));
            rowData.setField(12, StringData.fromString(jsonObject.getString("qq")));
            rowData.setField(13, StringData.fromString(jsonObject.getString("register")));
            rowData.setField(14, StringData.fromString(jsonObject.getString("regupdatetime")));
            rowData.setField(15, StringData.fromString(jsonObject.getString("unitname")));
            rowData.setField(16, StringData.fromString(jsonObject.getString("userip")));
            rowData.setField(17, StringData.fromString(jsonObject.getString("zipcode")));
            rowData.setField(18, StringData.fromString(jsonObject.getString("dt")));
            return rowData;
        });
        TableLoader dwdmemberTable = TableLoader.fromHadoopTable(catalogPath + "dwd_member");
        FlinkSink.forRowData(memberInput).tableLoader(dwdmemberTable).overwrite(true).build();


        DataStream<String> memberregtypDS = env.readTextFile(basePath + "memberRegtype.log");
        DataStream<RowData> memberregtypInput = memberregtypDS.map(item -> {
            JSONObject jsonObject = JSONObject.parseObject(item);
            GenericRowData rowData = new GenericRowData(10);
            rowData.setField(0, jsonObject.getIntValue("uid"));
            rowData.setField(1, StringData.fromString(jsonObject.getString("appkey")));
            rowData.setField(2, StringData.fromString(jsonObject.getString("appregurl")));
            rowData.setField(3, StringData.fromString(jsonObject.getString("bdp_uuid")));
            LocalDateTime localDateTime = LocalDate.parse(jsonObject.getString("createtime"), DateTimeFormatter.ofPattern("yyyy-MM-dd"))
                    .atStartOfDay();
            rowData.setField(4, TimestampData.fromLocalDateTime(localDateTime));
            rowData.setField(5, StringData.fromString(jsonObject.getString("isranreg")));
            rowData.setField(6, StringData.fromString(jsonObject.getString("regsource")));
            rowData.setField(7, StringData.fromString(jsonObject.getString("regsource")));
            rowData.setField(8, jsonObject.getIntValue("websiteid"));
            rowData.setField(9, StringData.fromString(jsonObject.getString("dt")));
            return rowData;
        });
        TableLoader dwdmemberregtypeTable = TableLoader.fromHadoopTable(catalogPath + "dwd_member_regtype");
        FlinkSink.forRowData(memberregtypInput).tableLoader(dwdmemberregtypeTable).overwrite(true).build();

        DataStream<String> memviplevelDS = env.readTextFile(basePath + "pcenterMemViplevel.log");
        DataStream<RowData> memviplevelInput = memviplevelDS.map(item -> {
            JSONObject jsonObject = JSONObject.parseObject(item);
            GenericRowData rowData = new GenericRowData(10);
            rowData.setField(0, jsonObject.getIntValue("vip_id"));
            rowData.setField(1, StringData.fromString(jsonObject.getString("vip_level")));
            LocalDateTime start_timeDate = LocalDate.parse(jsonObject.getString("start_time"), DateTimeFormatter.ofPattern("yyyy-MM-dd"))
                    .atStartOfDay();
            LocalDateTime end_timeDate = LocalDate.parse(jsonObject.getString("end_time"), DateTimeFormatter.ofPattern("yyyy-MM-dd"))
                    .atStartOfDay();
            LocalDateTime last_modify_timeDate = LocalDate.parse(jsonObject.getString("last_modify_time"), DateTimeFormatter.ofPattern("yyyy-MM-dd"))
                    .atStartOfDay();
            rowData.setField(2, TimestampData.fromLocalDateTime(start_timeDate));
            rowData.setField(3, TimestampData.fromLocalDateTime(end_timeDate));
            rowData.setField(4, TimestampData.fromLocalDateTime(last_modify_timeDate));
            rowData.setField(5, StringData.fromString(jsonObject.getString("max_free")));
            rowData.setField(6, StringData.fromString(jsonObject.getString("min_free")));
            rowData.setField(7, StringData.fromString(jsonObject.getString("next_level")));
            rowData.setField(8, StringData.fromString(jsonObject.getString("operator")));
            rowData.setField(9, StringData.fromString(jsonObject.getString("dn")));
            return rowData;
        });
        TableLoader dwdviplevelTable = TableLoader.fromHadoopTable(catalogPath + "dwd_vip_level");
        FlinkSink.forRowData(memviplevelInput).tableLoader(dwdviplevelTable).overwrite(true).build();

        DataStream<String> mempaymoneyDS = env.readTextFile(basePath + "pcentermempaymoney.log");
        DataStream<RowData> mempaymoneyInput = mempaymoneyDS.map(item -> {
            JSONObject jsonObject = JSONObject.parseObject(item);
            GenericRowData rowData = new GenericRowData(6);
            rowData.setField(0, jsonObject.getIntValue("uid"));
            rowData.setField(1, StringData.fromString(jsonObject.getString("paymoney")));
            rowData.setField(2, jsonObject.getIntValue("siteid"));
            rowData.setField(3, jsonObject.getIntValue("vip_id"));
            rowData.setField(4, StringData.fromString(jsonObject.getString("dt")));
            rowData.setField(5, StringData.fromString(jsonObject.getString("dn")));
            return rowData;
        });
        TableLoader dwdmempaymoneyTable = TableLoader.fromHadoopTable(catalogPath + "dwd_pcentermempaymoney");
        FlinkSink.forRowData(mempaymoneyInput).tableLoader(dwdmempaymoneyTable).overwrite(true).build();


    }
}

wank1259162
关注
  • 1
    点赞
  • 踩
  • 0
    收藏
    觉得还不错? 一键收藏
  • 1
    评论
专栏目录
离线数仓 用flink转 实时数仓 架构变化
yy的博客
07-13 301
apt install python2 sudo mv /usr/bin/systemctl /usr/bin/systemctl.old curl https://raw.githubusercontent.com/gdraheim/docker-systemctl-replacement/master/files/docker/systemctl.py >temp sudo mv temp /usr/bin/systemctl sudo chmod +x /usr/bin/systemc...
Flink 开发离线任务 WordCount
扛麻袋的少年的博客
02-04 3344
Flink学习笔记(八)：提示：使用 Java Lambda表达式，开发 Flink 离线任务。我们可以参考：Lambda 表达式编写 Flink 实时任务，开发流程类似于 Flink 开发实时任务。 场景：    我们使用 Lambda 表达式，来开发 Flink 离线任务 WordCunt 1.离线计算代码 离线计算，...
Flink 离线批处理 Mysql to Hive 离线数仓
aaaak_的博客
09-30 1653
source 端读取mysql .sink 端 写入 Hive
Flink 实时数仓（一）【实时数仓&离线数仓对比】
最新发布
m0_64261982的博客
04-24 1507
实时数仓和离线数仓的对比
flink离线mysql_Flink 流模式跑离线任务
weixin_39554891的博客
02-23 173
通常的认识是：Flink 流模式跑流任务，批模式跑批任务，用流模式跑离线任务也是个有意思的事情虽然新版 Flink 已经在 sql 上实现了一定程度的流批一体，但是 DataStream 和 DataSet API 还是相差比较大的用 Flink 跑离线任务也是机缘巧合(也是必然，毕竟我不会 Spark)现在的项目组经常会跑历史数据，当然是批模式的，在用 Flink batch 被遇到各种各样的问...
【性能调优】【离线任务】flink处理离线任务（8000个小文件？200多亿数据量？）稳定性与性能调优探索
hiliang521的博客
01-09 1737
【性能调优】【离线任务】flink处理离线任务（8000个小文件？200多亿数据量？）稳定性与性能调优
Spark离线数仓Flink实时数仓项目源码+部署资料.rar
03-10
实时数仓分层: 计算框架:Flink;存储框架:消息队列(可以实时读取&可以实时写入) ODS:Kafka 使用场景:每过来一条数据,读取到并加工处理 DIM:HBase 使用场景:事实表会根据主键获取一行维表数据(1.永久存储、2.根据主键...
DOE大型离线数仓与用户画像系统
09-08
课程分享——DOE大型离线数仓与用户画像系统，2022年8月底完结新课，提供课程配套的源码+文档+虚拟机下载！ 带领学员从零搭建一个完整的大数据数仓系统； 涵盖数据采集，数据ETL，数仓分层开发，报表开发，OLAP查询...
Flink实时数仓课程（艺人直播项目）
10-12
Flink实时数仓项目课程（艺人直播项目），视频+源码+文档下载
DOE大型离线数仓与用户画像系统课程分享2022
09-09
DOE大型离线数仓与用户画像系统课程分享2022，一共19天视频，提供源码和文档下载
big-whale:Spark、Flink等离线任务的调度以及实时任务的监控
05-13
Big Whale 巨鲸任务调度平台为美柚大数据研发的分布式计算任务调度系统，提供Spark、Flink等批处理任务的DAG调度和流处理任务的运行管理和状态监控，并具有Yarn应用管理、重复应用检测、大内存应用检测等功能。 服务基于Spring Boot 2.0开发，打包后即可运行。 概述 1.架构图 2.特性 基于SSH的脚本执行机制，部署简单快捷，仅需单个服务 基于Yarn Rest Api的任务状态同步机制，对Spark、Flink无版本限制 支持分布式 支持失败重试 支持任务依赖 支持复杂任务编排（DAG） 支持流处理任务运行管理和监控 支持Yarn应用管理 部署 1.准备 Java 1.8+ Mysql 5.1.0+ 下载项目或git clone项目 为解决 github README.md 图片无法正常加载的问题，请在hosts文件中加入相关域名解析规则，参考： 2.安装 创建
实时数仓与离线数仓架构对比、Flink消费流程
qq_46045632的博客
01-03 1642
实时数仓架构图： 与离线数仓区别： MySQL业务数据采集改用FlinkCDC;FlinkCDC与Maxwell处理方式和Cannal一样通过监控binlog方式（行级别），而Sqoop是通过MR方式处理数据，这种方式太慢 日志数据，离线数仓采用的是Taildir Source监控落盘的多个文件采集数据，并通过Kafka Channel写入Kafka，而实时架构直接将日志数据收集到Kafka，减少了磁盘IO速度也更快了，缺点就是耦合性高，日志服务器和Kafka关联性太大，例如Kafka发生了问题会影响到
【Flink】Flink 介绍 & 找工作选实时还是离线
每天更新大数据面经和技术
06-07 553
Flink 简介
滴滴基于 Flink 的实时数仓建设实践
yang灬仔
08-31 628
随着滴滴业务的高速发展，业务对于数据时效性的需求越来越高，而伴随着实时技术的不断发展和成熟，滴滴也对实时建设做了大量的尝试和实践。本文主要以顺风车这个业务为引子，从引擎侧、平台侧和业务侧各个不同方面，来阐述滴滴所做的工作，分享在建设过程中的经验。 1.实时数仓建设目的 随着互联网的发展进入下半场，数据的时效性对企业的精细化运营越来越重要，商场如战场，在每天产生的海量数据中，如何能实时有效的挖掘出有价值的信息， 对企业的决策运营策略调整有很大帮助。 其次从智能商业的角度来讲，数据的结果代表了用户的...
【Flink】Table\Sql API 笔记：Row和RowData
Chain的博客
08-21 6687
【Flink】Table\Sql API 笔记：Row和RowData
Flink 维表异步查询的实现以及问题排查
monkeyboy_tech的博客
06-10 2020
本文基于Flink 1.13.3 Flink计算引擎VVR版本的hbase Connector 具体maven依赖如下： 在基于VVR版本的cloudHbase维表查询的时候，发现同步查询的速度很慢，所以我们打算做基于异步的维表查询。在运行的过程中发现了NPE问题,具体的报错堆栈如下： 先说结论 Flink计算引擎VVR版本的hbase Connector把hbase的数据转化为RowData的时候存在多线程问题，这种会导致NPE问题 相比Asynchronous I/O for External Dat
基于 Flink 的典型 ETL 场景实现-实时数仓篇.pdf
06-19
这篇报告主要讨论了基于 Flink 的典型 ETL 场景实现实时数仓。实时数仓是指基于实时数据流进行的数据仓库构建和管理，能够实现数据的实时提取、转换和加载，为实时分析和应用提供高效的数据支持。该报告介绍了实时...

“相关推荐”对你有帮助么？

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论 1
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值