这里记录工作中遇到的一个问题,有一条sql在hive执行,在某些天执行异常。大概的sql结构如下
select * from
(select * from t_a where dt=*) a
left join t_b b
on a.column=b.column;
拷贝一份执行log作为记录
Execution log at: /tmp/ide/ide_20170118143800_ccc5d46f-cf9c-401e-a019-c859831e27cf.log
2017-01-18 14:38:10 Starting to launch local task to process map join; maximum memory = 1065484288
2017-01-18 14:38:14 Processing rows: 200000 Hashtable size: 199999 Memory usage: 69414544 percentage: 0.065
2017-01-18 14:38:15 Processing rows: 300000 Hashtable size: 299999 Memory usage: 99670168 percentage: 0.094
2017-01-18 14:38:16 Processing rows: 400000 Hashtable size: 399999 Memory usage: 131748832 percentage: 0.124
2017-01-18 14:38:17 Processing rows: 500000 Hashtable size: 499999 Memory usage: 162184848 percentage: 0.152
2017-01-18 14:38:17 Processing rows: 600000 Hashtable size: 599999 Memory usage: 191879808 percentage: 0.18
2017-01-18 14:38:18 Processing rows: 700000 Hashtable size: 699999 Memory usage: 222011048 percentage: 0.208
2017-