hive 查看历史job的执行sql

 

问题

线上hadoop有个hive sql执行较耗时，1hour 8min；收到运维预警

 

定位

查找job history 页面

http://node:8088/cluster/app/application_xxxx_xx

http://node:8088/proxy/application_xxxx_xx

http://node:19888/jobhistory/job/job_xxxx_xx

 

选取Local logs

 

hadoop-cmf-yarn-JOBHISTORY-node.log.out

 

从日志中查找到 job的conf.xml文件路径

hdfs://nameservice:8020/user/history/done/2020/02/01/001182/job_xxxx_xx_conf.xml

 

conf.xml中查找 hive.query.string

<name>hive.query.string</name>

<value> sql </value>

 

hadoop fs -text /user/history/done/2018/06/27/000008/job_1526892856952_8007_conf.xml

 

解决

• 全量修改为增量
• set 参数限制mapper reduce 数量

 

set mapred.max.split.size=1024000000;
set mapred.min.split.size.per.node=1024000000;
set mapred.min.split.size.per.rack=1024000000;

set mapred.reduce.tasks=10;

 

参考

http://blog.sina.com.cn/s/blog_6dd718930102xrhl.html