大数据hive篇--hive去重

在路上的小y

已于 2022-09-27 16:52:51 修改

阅读量2.3k

点赞数 1

分类专栏：大数据文章标签：大数据 hive hadoop

于 2022-09-20 17:36:05 首次发布

本文链接：https://blog.csdn.net/qq_43709558/article/details/126957933

版权

文章目录

1.使用distinct

问题：
每个app下只保留一个用户
案例：

spark-sql> with test1 as
         > (select 122 as userid,100024 as apptypeid
         > union all
         > select 123 as userid,100024 as apptypeid
         > union all
         > select 123 as userid,100024 as apptypeid)
         > select 
         >   distinct userid,apptypeid
         > from test1;

2.使用group by

问题：
每个app下只保留一个用户
案例：

spark-sql> with test1 as
         > (select 122 as userid,100024 as apptypeid
         > union all
         > select 123 as userid,100024 as apptypeid
         > union all
         > select 123 as userid,100024 as apptypeid)
         > select 
         >   userid,
         >   apptypeid
         > from 
         > (select 
         >   userid,
         >   apptypeid
         > from test1) t1
         > group by userid,apptypeid;