由hbase.client.scanner.caching参数引发的血案

最新推荐文章于 2022-10-24 21:24:17 发布

rzhzhz

最新推荐文章于 2022-10-24 21:24:17 发布

阅读量9.1k

点赞数 1

分类专栏： hive hadoop 分布式云计算 Hbase 文章标签： mapreduce hadoop cache hbase sql 任务

本文链接：https://blog.csdn.net/rzhzhz/article/details/7536285

版权

在Hadoop 0.20.203.0和Hbase 0.90.3环境下，执行Hive SQL查询时遇到任务假死问题。经过排查发现，问题源于`hbase.client.scanner.caching`参数未在客户端生效，导致每次从Hbase扫描数据耗时70+ms。通过在HiveHBaseTableInputFormat类中手动设置Scan的cache值，并确保在客户端配置中设置该参数，最终解决了问题，提高了MapReduce任务的处理速度。

摘要由CSDN通过智能技术生成

环境描述

Hadoop 0.20.203.0
Hbase 0.90.3
Hive 0.80.1

问题描述

前几天，在HIVE执行SQL查询的时候出现了一个很奇怪的问题：就是每个SQL（涉及到MapReduce的SQL任务）在执行到某个百分比的时候，整个JOB会出现假死的情况。

2012-04-28 18:22:33,661 Stage-1 map = 0%, reduce = 0%
2012-04-28 18:22:59,760 Stage-1 map = 25%, reduce = 0%
2012-04-28 18:23:04,782 Stage-1 map = 38%, reduce = 0%
2012-04-28 18:23:07,796 Stage-1 map = 50%, reduce = 0%
2012-04-28 18:23:08,801 Stage-1 map = 50%, reduce = 8%
2012-04-28 18:23:17,839 Stage-1 map = 50%, reduce = 17%
2012-04-28 18:23:19,848 Stage-1 map = 63%, reduce = 17%
2012-04-28 18:23:32,909 Stage-1 map = 63%, reduce = 21%
2012-04-28 18:23:57,017 Stage-1 map = 75%, reduce = 21%
2012-04-28 18:24:09,075 Stage-1 map = 75%, reduce = 25%
2012-04-28 18:25:09,397 Stage-1 map = 75%, reduce = 25%
2012-04-28 18:26:09,688 Stage-1 map = 75%, reduce = 25%
2012-04-28 18:27:09,980 Stage-1 map = 75%, reduce = 25%
2012-04-28 18:28:10,262 Stage-1 map = 75%, reduce = 25%
2012-04-28 18:29:10,522 Stage-1 map = 75%, reduce = 25%
2012-04-28 18:30:10,742 Stage-1 map = 75%, reduce = 25%
2012-04-28 18:31:10,985 Stage-1 map = 75%, reduce = 25%
2012-04-28 18:32:11,238 Stage-1 map = 75%, reduce = 25%
2012-04-28 18:33:11,467 Stage-1 map = 75%, reduce = 25%
2012-04-28 18:34:11,731 Stage-1 map = 75%, reduce = 25%
2012-04-28 18:35:11,968 Stage-1 map = 75%, reduce = 25%
2012-04-28 18:36:12,213 Stage-1 map = 75%, reduce = 25%
2012-04-28 18:37:12,508 Stage-1 map = 75%, red