CC00070.pbpositions——|Hadoop&PB级数仓.V04|——|PB数仓.v04|元数据管理工具Atlas|电商业务集成|血缘关系|

yanqi_vip

已于 2022-04-12 18:09:42 修改

阅读量367

点赞数

分类专栏： bigdatav014——PB离线数仓文章标签：大数据 hive hadoop mysql python

于 2022-04-11 19:20:00 首次发布

不予转载

本文链接：https://blog.csdn.net/yanqi_vip/article/details/124114479

版权

本文介绍了如何将电商业务集成到Hadoop PB级数仓中，详细阐述了业务建表过程、数据加载步骤，并探讨了使用Atlas进行元数据管理和血缘关系追踪的方法。

摘要由CSDN通过智能技术生成

一、与电商业务集成

### --- 电商业务集成环境准备

~~~     开发(建库、建表) => 导入数据 => 执行Hive脚本
~~~     导入Hive的血缘关系

二、电商业务建表

### --- 电商业务建表语句(可省略)：
~~~     创建DataBases;

CREATE DATABASE ODS;
CREATE DATABASE DIM;
CREATE DATABASE DWD;
CREATE DATABASE DWS;
CREATE DATABASE ADS;

### --- 创建ODS表

DROP TABLE IF EXISTS `ods.ods_trade_orders`;

CREATE EXTERNAL TABLE `ods.ods_trade_orders`(
`orderid` int,
`orderno` string,
`userid` bigint,
`status` tinyint,
`productmoney` decimal(10,0),
`totalmoney` decimal(10,0),
`paymethod` tinyint,
`ispay` tinyint,
`areaid` int,
`tradesrc` tinyint,
`tradetype` int,
`isrefund` tinyint,
`dataflag` tinyint,
`createtime` string,
`paytime` string,
`modifiedtime` string)
COMMENT '订单表'
PARTITIONED BY (`dt` string)
row format delimited fields terminated by ','
location '/data/yanqidw/data/trade.db/orders/';

DROP TABLE IF EXISTS `ods.ods_trade_order_product`;

CREATE EXTERNAL TABLE `ods.ods_trade_order_product`(
`id` string,
`orderid` decimal(10,2),
`productid` string,
`productnum` string,
`productprice` string,
`money` string,
`extra` string,
`createtime` string)
COMMENT '订单明细表'
PARTITIONED BY (`dt` string)
row format delimited fields terminated by ','
location '/data/yanqidw/data/trade.db/order_product/';

DROP TABLE IF EXISTS `ods.ods_trade_product_info`;

CREATE EXTERNAL TABLE `ods.ods_trade_product_info`(
`productid` bigint,
`productname` string,
`shopid` string,
`price` decimal(10,0),
`issale` tinyint,
`status` tinyint,
`categoryid` string,
`createtime` string,
`modifytime` string)
COMMENT '产品信息表'
PARTITIONED BY (`dt` string)
row format delimited fields terminated by ',&