hive合并小文件

hive合并小文件

1. 创建备用表

 create table new_table like old_table;

2. 设置合并参数，再将原表数据插入到备用表

SET hive.merge.mapfiles = true;
SET hive.merge.mapredfiles = true;
SET hive.merge.size.per.task = 256000000;
SET hive.merge.smallfiles.avgsize = 134217728;
SET hive.exec.compress.output = true;
SET parquet.compression = snappy;
SET hive.exec.dynamic.partition.mode = nonstrict;
SET hive.exec.dynamic.partition = true;

insert overwrite table new_table partition(dt) select * from old_table;

3.确认二表数据一致后，将备用表的表名修改为原表的表名。废除旧表

drop  table  old_table;

alter table new_table rename to old_table  ;

4.可运用hdfs -ls 命令查看该表下小文件的已合并。