关于hvie对join优化的一个隐藏的坑

最新推荐文章于 2024-05-10 14:54:23 发布

nieyanshun_me

最新推荐文章于 2024-05-10 14:54:23 发布

阅读量2.9k

点赞数 1

分类专栏： hive 文章标签： hive

本文链接：https://blog.csdn.net/nieyanshun_me/article/details/54601492

版权

这里记录工作中遇到的一个问题，有一条sql在hive执行，在某些天执行异常。大概的sql结构如下

    select * from 
        (select * from t_a where dt=*) a
    left join t_b b
     on a.column=b.column;

拷贝一份执行log作为记录

Execution log at: /tmp/ide/ide_20170118143800_ccc5d46f-cf9c-401e-a019-c859831e27cf.log
2017-01-18 14:38:10 Starting to launch local task to process map join;  maximum memory = 1065484288
2017-01-18 14:38:14 Processing rows:    200000  Hashtable size: 199999  Memory usage:   69414544    percentage: 0.065
2017-01-18 14:38:15 Processing rows:    300000  Hashtable size: 299999  Memory usage:   99670168    percentage: 0.094
2017-01-18 14:38:16 Processing rows:    400000  Hashtable size: 399999  Memory usage:   131748832   percentage: 0.124
2017-01-18 14:38:17 Processing rows:    500000  Hashtable size: 499999  Memory usage:   162184848   percentage: 0.152
2017-01-18 14:38:17 Processing rows:    600000  Hashtable size: 599999  Memory usage:   191879808   percentage: 0.18
2017-01-18 14:38:18 Processing rows:    700000  Hashtable size: 699999  Memory usage:   222011048   percentage: 0.208
2017-