1、问题描述:
在hive的shell端执行:select count(*) from student;出现了下面问题(一直卡着):
hive> select count(*) from student;
WARNING: Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
Query ID = root_20190712235937_41a66280-b28b-4414-8a68-4e52e243bff3
Total jobs = 1
Launching Job 1 out of 1
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapreduce.job.reduces=<number>
Starting Job = job_1562982531171_0001, Tracking URL = http://cmaster:8088/proxy/application_1562982531171_0001/
Kill Command = /opt/softWare/hadoop/hadoop-2.7.3/bin/hadoop job -kill job_1562982531171_0001
Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 1
2019-07-13 00:00:21,337 Stage-1 map = 0%, reduce = 0%
2019-07-13 00:01:21,531 Stage-1 map = 0%, reduce = 0%
2019-07-13 00:02:22,523 Stage-1 map = 0%, reduce = 0%
2019-07-13 00:03:23,383 Stage-1 map = 0%, reduce = 0%
2019-07-13 00:04:23,894 Stage-1 map = 0%, reduce = 0%
2019-07-13 00:05:24,817 Stage-1 map = 0%, reduce = 0%
2019-07-13 00:06:25,394 Stage-1 map = 0%, reduce = 0%
2019-07-13 00:07:26,368 Stage-1 map = 0%, reduce = 0%
2019-07-13 00:08:26,770 Stage-1 map = 0%, reduce = 0%
2019-07-13 00:09:27,813 Stage-1 map = 0%, reduce = 0%
2019-07-13 00:10:28,492 Stage-1 map = 0%, reduce = 0%
2019-07-13 00:11:29,124 Stage-1 map = 0%, reduce = 0%
......
2、解决办法:
(1)进入hadoop安装目录下,修改yarn-site.xml
cd /opt/softWare/hadoop/hadoop-2.7.3/etc/hadoop
vim yarn-site.xml
将
<name>yarn.nodemanager.resource.memory-mb</name>
<value>2048</value>
</property>
修改为:
<name>yarn.nodemanager.resource.memory-mb</name>
<value>4096</value>
</property>
(2)重启yarn服务:
cd /opt/softWare/hadoop/hadoop-2.7.3/sbin/
[root@cmaster sbin]# ./stop-yarn.sh
[root@cmaster sbin]# ./start-yarn.sh
3、完美解决:
hive> select count(*) from student;
WARNING: Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
Query ID = root_20190713002124_69d40918-a6b3-4755-ae76-e45e695cfeac
Total jobs = 1
Launching Job 1 out of 1
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapreduce.job.reduces=<number>
Starting Job = job_1563002427699_0001, Tracking URL = http://cmaster:8088/proxy/application_1563002427699_0001/
Kill Command = /opt/softWare/hadoop/hadoop-2.7.3/bin/hadoop job -kill job_1563002427699_0001
Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 1
2019-07-13 00:21:53,647 Stage-1 map = 0%, reduce = 0%
2019-07-13 00:22:14,111 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 2.67 sec
2019-07-13 00:22:33,107 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 5.67 sec
MapReduce Total cumulative CPU time: 5 seconds 670 msec
Ended Job = job_1563002427699_0001
MapReduce Jobs Launched:
Stage-Stage-1: Map: 1 Reduce: 1 Cumulative CPU: 5.67 sec HDFS Read: 7660 HDFS Write: 101 SUCCESS
Total MapReduce CPU Time Spent: 5 seconds 670 msec
OK
3
Time taken: 70.166 seconds, Fetched: 1 row(s)