ETL hive update 之 deltamerge 优化

ETL hive update 之 deltamerge 优化

• full join 横向join ，不能map join 走shuffle
• row_number() over ( partition by 主键 order by $flag desc) rank ... where rank =1 ，走shufle

select 
  id, 
  order_datekey, 
  f_procurement_order, 
from 
  (
    select 
      id, 
      order_datekey, 
      f_procurement_order, 
    
      row_number() over (
        partition by id 
        order by 
          b_flag_i desc
      ) rank 
    from 
      (
        select 
          id, 
          order_datekey, 
          f_procurement_order, 
          0 b_flag_i 
        from 
          ods_pms_procurement_order_item_hm old 
        WHERE 
          c_t >= 1479916800 
          or u_t >= 1479916800 
        union all 
        select 
          id, 
          order_datekey, 
          f_procurement_order, 
          1 b_flag_i 
        from 
          ods_pms_procurement_order_item_hm_delta_64124FEADBFA9720 new
      ) t
  ) st 
where 
  rank = 1;

• 差集 + 并集方式 效率最高 前提是增量数据较少，要不也要走shuffle

# semi_1 数据 id , name
1       jx
2       gj

# semi_2数据id, age
1       28
3       30

select a.id,a.name from semi_1 a left anti join semi_2 b on a.id = b.id;
left anti join 是以左表为主，如果join上就返回null，否则返回左表数据。
2       gj

下图a代表完整old 全集，b代表完整new全集，c代表a与b join 上的交集部分（比如id相等的部分）所以思路就是（a-c）+b实现hive 的update

SET hive.mapred.mode=nonstrict;
INSERT overwrite TABLE $target.table 
SELECT 
  $stream.format 
FROM 
  $target.table old left anti 
  join ($delta) new on $stream.unique_keys
UNION ALL 
SELECT 
  $stream.format 
FROM 
  $target.table ;

fields = 'id,name'

new = 'new'
old = 'old'
and_str = ' AND '
cmd = []
for field in fields.split(','):
    str = old + '.' + field + ' = ' + new + '.' + field
    cmd.append(str)

print and_str.join(cmd)

posted on 2019-03-15 20:25 姜小嫌 阅读( ...) 评论( ...) 编辑 收藏

转载于:https://www.cnblogs.com/jiangxiaoxian/p/10539329.html