hive小文件合并机制_hive优化之自动合并输出的小文件

1.先在hive-site.xml中设置小文件的标准.

hive.merge.smallfiles.avgsize

536870912

When the average output file size of a job is less than this number, Hive will start an additional map-reduce job to merge the output files into bigger files. This is only done for map-only jobs if hive.merge.mapfiles is true, and for map-reduce jobs if hive.merge.mapredfiles is true.

2.为只有map的mapreduce的输出并合并小文件.

hive.merge.mapfiles

true

Merge small files at the end of a map-only job

3.为含有reduce的mapreduce的输出并合并小文件.

hive.merge.mapredfiles

true

Merge small files at the end of a map-reduce job

更多相关教程请访问 MySQL视频教程

本条技术文章来源于互联网，如果无意侵犯您的权益请点击此处反馈版权投诉

本文系统来源：php中文网

TAG标签：hive优化