tez得文件生成个数与什么有关？

cclovezbf

已于 2023-05-15 15:12:12 修改

阅读量710

点赞数

分类专栏： hive 文章标签： hive tez

于 2023-04-07 17:04:55 首次发布

本文链接：https://blog.csdn.net/cclovezbf/article/details/129995833

版权

hive 专栏收录该内容

49 篇文章 10 订阅

订阅专栏

之前研究过hive on spark，最后的文件生成个数
hive-mapJoin和skewJoin-CSDN博客
现在在cdp了使用的是hive on tez。也在学习下这个文件的生成个数与什么有关？

以该表为例有11个文件每个差不多60M左右总共620M左右。

我们先想下哪些参数和生成的文件数有关？

set hive.tez.cpu.vcores --好像是tez处理时用的core 类似spark的core 默认没有

set mapred.reduce.tasks=10 --reduce的个数默认-1

set hive.exec.reducers.bytes.per.reducer=67108864 --默认64M

set hive.merge.tezfiles=false --合并小文件默认false

-- set mapreduce.reduce.cpu.vcores=1 --reduce使用的core 默认1 这个参数在set找得到但是官网和hive的文档里找不到，不做考虑

distribute by

order by

来点参数说明

hive.tez.cpu.vcores

Default Value: -1
Added In: Hive 0.14.0 with HIVE-8452

By default Tez will ask for however many CPUs MapReduce is configured to use per container. This can be used to overwrite the default.

mapred.reduce.tasks

Default Value: -1
Added In: Hive 0.1.0

The default number of reduce tasks per job. Typically set to a prime close to the number of available hosts. Ignored when mapred.job.tracker is "local". Hadoop set this to 1 by default, whereas Hive uses -1 as its default value. By setting this property to -1, Hive will automatically figure out what should be the number of reducers.

hive.exec.reducers.bytes.per.reducer

Default Value: 1,000,000,000 prior to Hive 0.14.0; 256 MB (256,000,000) in Hive 0.14.0 and later
Added In: Hive 0.2.0; default changed in 0.14.0 with HIVE-7158 (and HIVE-7917)

Size per reducer. The default in Hive 0.14.0 and earlier is 1 GB, that is, if the input size is 10 GB then 10 reducers will be used. In Hive 0.14.0 and later the default is 256 MB, that is, if the input size is 1 GB then 4 reducers will be used.

hive.merge.tezfiles

Default Value: false
Added In: Hive 0.13.0 with HIVE-6498 and HIVE-6360

Merge small files at the end of a Tez DAG.

经过我各种测试各种参数修改发现最后的结果都是一样的，都是11个。。。。我就觉得不对劲了。那里出错了呢？怎么找呢？

看yarn app

我的appId是1377

而我都已经设置了。

set hive.exec.reducers.bytes.per.reducer=134217728
set mapred.reduce.tasks=5
set hive.merge.tezfiles=true
set hive.tez.cpu.vcores=4

insert overwrite table odsmdmdata.sms_vendor_manage_t
select * from odsmdmdata.sms_vendor_manage_t