hive理解streamtable使用

最新推荐文章于 2024-04-26 09:58:25 发布

slx_2011

最新推荐文章于 2024-04-26 09:58:25 发布

阅读量6.5k

点赞数 1

分类专栏： hadoop那点事文章标签： hive join 优化 streamtable

本文链接：https://blog.csdn.net/slx_2011/article/details/20039029

版权

hadoop那点事专栏收录该内容

61 篇文章 0 订阅

订阅专栏

今天看别人的博客，发现streamtable这个东西，作者描述是：

将大表放在JION的右边，这是就需要指定使用/*+ STREAMTABLE(..) */：

 
 hive> SELECT /*+ STREAMTABLE(b) */ a.val, b.val, c.val FROM a JOIN b  
    > ON (a.key = b.key1) JOIN c将大表放在JION的右边，这是就需要指定使用/*+ STREAMTABLE(..) */：


hive> SELECT /*+ STREAMTABLE(b) */ a.val, b.val, c.val FROM a JOIN b  
    > ON (a.key = b.key1) JOIN c ON (c.key = b.key1) ON (c.key = b.key1)

 
 
  
  

 
 
 
 
  
  

 
 
 
 
  
  

 
 
 
 
  
  有点懵懂，看完另一个哥们写的才若有所悟
 
 
 
 
  
  

 
 
 
 
  
  

 
 
 
 
  
  
   
   From my understanding, when you have the join happening in map or reduce, the values corresponding to a key from all all table's except one (if two tables are involved in join on same key, then just one table here) are buffered in memory and the left out one is streamed. Usually it is the largest table to be streamed, else the larger data can go into the memory(buffer) and create OOM errors.
  
  
  
  
   
   

  
  
  
  
   
   This stream table hint is used to specify which table to be streamed. By default it is the table that comes on the right is streamed and the other is buffered. But if you wan't  other  than right table to be streamed you go for this hint.
  
  
  
  
   
   

  
  
  
  
   
   If you are joining more tables on different keys, then for every join set just specify the larger table on the right of ON condition. No need of stream table hint here.