hive表数据去重

根据user name查重
SELECT test_user_name, count(*) c from default.test GROUP BY test_user_name HAVING c> 1;
根据id查重
SELECT id ,count(*) c FROM default.test GROUP BY id HAVING c >1;
去重：注意min用法，取相同数据的最小id，去重的重点
min:使用group by 分组后取分组内最小id，以便去掉其它重复数据
INSERT OVERWRITE table default.test partition(test_data_source) select * from default.test where id in ( select min(id) as id from default.test group by test_user_name) ;