hive合并小文件
- 创建备用表
create table new_table like old_table;
- 设置合并参数,再将原表数据插入到备用表
SET hive.merge.mapfiles = true;
SET hive.merge.mapredfiles = true;
SET hive.merge.size.per.task = 256000000;
SET hive.merge.smallfiles.avgsize = 134217728;
SET hive.exec.compress.output = true;
SET parquet.compression = snappy;
SET hive.exec.dynamic.partition.mode = nonstrict;
SET hive.exec.dynamic.partition = true;
insert overwrite table new_table partition(dt) select * from old_table;
3.确认二表数据一致后,将备用表的表名修改为原表的表名。废除旧表
drop table old_table;
alter table new_table rename to old_table ;
4.可运用hdfs -ls 命令查看该表下小文件的已合并。