大数据hive篇--hive去重

原创

已于 2022-09-27 16:52:51 修改 · 2.4k 阅读

5 ·

CC 4.0 BY-SA版权

文章标签：

#大数据 #hive #hadoop

于 2022-09-20 17:36:05 首次发布

文章目录

1.使用distinct

问题：
每个app下只保留一个用户
案例：

spark-sql> with test1 as
         > (select 122 as userid,100024 as apptypeid
         > union all
         > select 123 as userid,100024 as apptypeid
         > union all
         > select 123 as userid,100024 as apptypeid)
         > select 
         >   distinct userid,apptypeid
         > from test1;

2.使用group by

问题：
每个app下只保留一个用户
案例：

spark-sql> with test1 as
         > (select 122 as userid,100024 as apptypeid
         > union all
         > select 123 as userid,100024 as apptypeid
         > union all
         > select 123 as userid,100024 as apptypeid)
         > select 
         >   userid,
         >   apptypeid
         > from 
         > (select 
         >   userid,
         >   apptypeid
         > from test1) t1
         > group by userid,apptypeid;