Hive跑MapReduce报错【已解决】_[stage-8:mapredlocal] in serial mode-CSDN博客

本文链接：https://blog.csdn.net/Yushuzii/article/details/102570218

1.报错日志

2019-10-14 13:05:18,815 INFO  [main]: ql.Driver (SessionState.java:printInfo(1087)) - Query ID = tomcat_20191014130505_85923f08-f137-47a4-9516-c26144f3bfda
2019-10-14 13:05:18,815 INFO  [main]: ql.Driver (SessionState.java:printInfo(1087)) - Total jobs = 3
2019-10-14 13:05:18,836 INFO  [main]: ql.Driver (Driver.java:launchTask(2052)) - Starting task [Stage-17:MAPREDLOCAL] in serial mode
2019-10-14 13:05:18,839 INFO  [main]: mr.MapredLocalTask (MapredLocalTask.java:executeInChildVM(175)) - Generating plan file file:/tmp/tomcat/9bcc1db2-89c7-4a4b-bccf-512104e09444/hive_2019-10-14_13-05-14_002_5277827082352258207-1/-local-10008/plan.xml
2019-10-14 13:05:19,300 INFO  [main]: mr.MapredLocalTask (MapredLocalTask.java:executeInChildVM(322)) - Executing: /usr/local/cloudera-manager/cloudera/parcels/CDH-5.15.0-1.cdh5.15.0.p0.21/lib/hadoop/bin/hadoop jar /usr/local/cloudera-manager/cloudera/parcels/CDH-5.15.0-1.cdh5.15.0.p0.21/jars/hive-exec-1.1.0-cdh5.15.0.jar org.apache.hadoop.hive.ql.exec.mr.ExecDriver -libjars file:///usr/local/cloudera-manager/cloudera/parcels/CDH-5.15.0-1.cdh5.15.0.p0.21/lib/hbase/hbase-common.jar,file:/usr/local/cloudera-manager/cloudera/parcels/CDH-5.15.0-1.cdh5.15.0.p0.21/lib/hive/auxlib/hive-exec-1.1.0-cdh5.15.0-core.jar,file:///usr/local/cloudera-manager/cloudera/parcels/CDH-5.15.0-1.cdh5.15.0.p0.21/lib/hbase/hbase-hadoop-compat.jar,file:///usr/local/cloudera-manager/cloudera/parcels/CDH-5.15.0-1.cdh5.15.0.p0.21/lib/hbase/lib/htrace-core-3.2.0-incubating.jar,file:///usr/local/cloudera-manager/cloudera/parcels/CDH-5.15.0-1.cdh5.15.0.p0.21/lib/hbase/hbase-client.jar,file:///usr/share/java/mysql-connector-java.jar,file:///usr/local/cloudera-manager/cloudera/parcels/CDH-5.15.0-1.cdh5.15.0.p0.21/lib/hbase/hbase-hadoop2-compat.jar,file:///usr/local/cloudera-manager/cloudera/parcels/CDH-5.15.0-1.cdh5.15.0.p0.21/lib/hbase/lib/htrace-core4-4.0.1-incubating.jar,file:///usr/local/cloudera-manager/cloudera/parcels/CDH-5.15.0-1.cdh5.15.0.p0.21/lib/hbase/hbase-server.jar,file:///usr/local/cloudera-manager/cloudera/parcels/CDH-5.15.0-1.cdh5.15.0.p0.21/lib/hive/lib/hive-hbase-handler-1.1.0-cdh5.15.0.jar,file:///usr/local/cloudera-manager/cloudera/parcels/CDH-5.15.0-1.cdh5.15.0.p0.21/lib/hbase/hbase-protocol.jar,file:/usr/local/cloudera-manager/cloudera/parcels/CDH-5.15.0-1.cdh5.15.0.p0.21/lib/hive/auxlib/hive-exec-core.jar,file:///usr/local/cloudera-manager/cloudera/parcels/CDH-5.15.0-1.cdh5.15.0.p0.21/lib/hbase/lib/htrace-core.jar  -localtask -plan file:/tmp/tomcat/9bcc1db2-89c7-4a4b-bccf-512104e09444/hive_2019-10-14_13-05-14_002_5277827082352258207-1/-local-10008/plan.xml   -jobconffile file:/tmp/tomcat/9bcc1db2-89c7-4a4b-bccf-512104e09444/hive_2019-10-14_13-05-14_002_5277827082352258207-1/-local-10009/jobconf.xml
2019-10-14 13:05:19,362 INFO  [main-LocalTask-MAPREDLOCAL-stderr]: mr.MapredLocalTask (LogRedirector.java:run(65)) - Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512M; support was removed in 8.0
2019-10-14 13:05:40,711 INFO  [main-LocalTask-MAPREDLOCAL-stderr]: mr.MapredLocalTask (LogRedirector.java:run(65)) - 2019-10-14 01:05:40	Processing rows:	200000	Hashtable size:	199999	Memory usage:	105744560	percentage:	0.055
2019-10-14 13:05:47,635 INFO  [main-LocalTask-MAPREDLOCAL-stderr]: mr.MapredLocalTask (LogRedirector.java:run(65)) - 2019-10-14 01:05:47	Processing rows:	400000	Hashtable size:	399999	Memory usage:	364921464	percentage:	0.191
2019-10-14 13:05:54,620 INFO  [main-LocalTask-MAPREDLOCAL-stderr]: mr.MapredLocalTask (LogRedirector.java:run(65)) - 2019-10-14 01:05:54	Processing rows:	600000	Hashtable size:	599999	Memory usage:	269948592	percentage:	0.141
2019-10-14 13:06:04,205 INFO  [main-LocalTask-MAPREDLOCAL-stderr]: mr.MapredLocalTask (LogRedirector.java:run(65)) - 2019-10-14 01:06:04	Processing rows:	900000	Hashtable size:	899999	Memory usage:	298878712	percentage:	0.157
2019-10-14 13:06:18,823 INFO  [main-LocalTask-MAPREDLOCAL-stderr]: mr.MapredLocalTask (LogRedirector.java:run(65)) - 2019-10-14 01:06:18	Processing rows:	1300000	Hashtable size:	1299999	Memory usage:	462124912	percentage:	0.242
2019-10-14 13:06:27,881 INFO  [main-LocalTask-MAPREDLOCAL-stderr]: mr.MapredLocalTask (LogRedirector.java:run(65)) - 2019-10-14 01:06:24	Processing rows:	1500000	Hashtable size:	1499999	Memory usage:	733931168	percentage:	0.384
2019-10-14 13:06:33,541 INFO  [main-LocalTask-MAPREDLOCAL-stderr]: mr.MapredLocalTask (LogRedirector.java:run(65)) - 2019-10-14 01:06:33	Processing rows:	1800000	Hashtable size:	1799999	Memory usage:	814752808	percentage:	0.427
2019-10-14 13:06:42,294 INFO  [main-LocalTask-MAPREDLOCAL-stderr]: mr.MapredLocalTask (LogRedirector.java:run(65)) - 2019-10-14 01:06:42	Processing rows:	2000000	Hashtable size:	1999999	Memory usage:	743289136	percentage:	0.389
2019-10-14 13:06:45,241 INFO  [main-LocalTask-MAPREDLOCAL-stderr]: mr.MapredLocalTask (LogRedirector.java:run(65)) - 2019-10-14 01:06:45	Processing rows:	2100000	Hashtable size:	2099999	Memory usage:	879025096	percentage:	0.46
2019-10-14 13:06:51,695 INFO  [main-LocalTask-MAPREDLOCAL-stderr]: mr.MapredLocalTask (LogRedirector.java:run(65)) - 2019-10-14 01:06:51	Processing rows:	2300000	Hashtable size:	2299999	Memory usage:	903924936	percentage:	0.474
2019-10-14 13:07:04,304 INFO  [main-LocalTask-MAPREDLOCAL-stderr]: mr.MapredLocalTask (LogRedirector.java:run(65)) - 2019-10-14 01:07:04	Processing rows:	2700000	Hashtable size:	2699999	Memory usage:	1006784552	percentage:	0.527
2019-10-14 13:07:12,424 INFO  [main-LocalTask-MAPREDLOCAL-stderr]: mr.MapredLocalTask (LogRedirector.java:run(65)) - 2019-10-14 01:07:12	Processing rows:	3000000	Hashtable size:	2999999	Memory usage:	1047621488	percentage:	0.549
2019-10-14 13:07:18,192 INFO  [main-LocalTask-MAPREDLOCAL-stderr]: mr.MapredLocalTask (LogRedirector.java:run(65)) - 2019-10-14 01:07:18	Processing rows:	3200000	Hashtable size:	3199999	Memory usage:	1171750992	percentage:	0.614
2019-10-14 13:07:24,018 INFO  [main-LocalTask-MAPREDLOCAL-stderr]: mr.MapredLocalTask (LogRedirector.java:run(65)) - 2019-10-14 01:07:24	Processing rows:	3400000	Hashtable size:	3399999	Memory usage:	1103519880	percentage:	0.578
2019-10-14 13:07:42,015 INFO  [main-LocalTask-MAPREDLOCAL-stderr]: mr.MapredLocalTask (LogRedirector.java:run(65)) - 2019-10-14 01:07:42	Processing rows:	3900000	Hashtable size:	3899999	Memory usage:	1401950664	percentage:	0.734
2019-10-14 13:07:50,611 INFO  [main-LocalTask-MAPREDLOCAL-stderr]: mr.MapredLocalTask (LogRedirector.java:run(65)) - 2019-10-14 01:07:50	Processing rows:	4200000	Hashtable size:	4199999	Memory usage:	1442872208	percentage:	0.756
2019-10-14 13:07:56,961 INFO  [main-LocalTask-MAPREDLOCAL-stderr]: mr.MapredLocalTask (LogRedirector.java:run(65)) - 2019-10-14 01:07:56	Processing rows:	4400000	Hashtable size:	4399999	Memory usage:	1543953064	percentage:	0.809
2019-10-14 13:08:10,145 INFO  [main-LocalTask-MAPREDLOCAL-stderr]: mr.MapredLocalTask (LogRedirector.java:run(65)) - 2019-10-14 01:08:10	Processing rows:	4700000	Hashtable size:	4699999	Memory usage:	1584757192	percentage:	0.83
2019-10-14 13:08:25,193 INFO  [main-LocalTask-MAPREDLOCAL-stderr]: mr.MapredLocalTask (LogRedirector.java:run(65)) - 2019-10-14 01:08:25	Processing rows:	4900000	Hashtable size:	4899999	Memory usage:	1529279360	percentage:	0.801
2019-10-14 13:08:37,106 INFO  [main-LocalTask-MAPREDLOCAL-stderr]: mr.MapredLocalTask (LogRedirector.java:run(65)) - 2019-10-14 01:08:37	Processing rows:	5100000	Hashtable size:	5099999	Memory usage:	1605237048	percentage:	0.841
2019-10-14 13:08:58,762 INFO  [main-LocalTask-MAPREDLOCAL-stderr]: mr.MapredLocalTask (LogRedirector.java:run(65)) - 2019-10-14 01:08:58	Processing rows:	5300000	Hashtable size:	5299999	Memory usage:	1661488104	percentage:	0.87
2019-10-14 13:11:53,183 INFO  [main-LocalTask-MAPREDLOCAL-stderr]: mr.MapredLocalTask (LogRedirector.java:run(65)) - Hive Runtime Error: Map local work exhausted memory
2019-10-14 13:12:16,237 ERROR [main]: exec.Task (SessionState.java:printError(1110)) - Execution failed with exit status: 3
2019-10-14 13:12:16,237 ERROR [main]: exec.Task (SessionState.java:printError(1110)) - Obtaining error information
2019-10-14 13:12:16,238 ERROR [main]: exec.Task (SessionState.java:printError(1110)) - 
Task failed!
Task ID:
  Stage-17

Logs:

2019-10-14 13:12:16,238 ERROR [main]: exec.Task (SessionState.java:printError(1110)) - /tmp/tomcat/hive.log
2019-10-14 13:12:16,239 ERROR [main]: mr.MapredLocalTask (MapredLocalTask.java:executeInChildVM(372)) - Execution failed with exit status: 3
2019-10-14 13:12:16,290 ERROR [main]: ql.Driver (SessionState.java:printError(1110)) - FAILED: Execution Error, return code 3 from org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask
2019-10-14 13:12:16,291 INFO  [main]: ql.Driver (Driver.java:execute(1960)) - Completed executing command(queryId=tomcat_20191014130505_85923f08-f137-47a4-9516-c26144f3bfda); Time taken: 417.479 seconds

2.解决方案

上面的报错，有一句是 Hive Runtime Error: Map local work exhausted memory，可以看出是内存耗尽了，做一下优化

--是否自动转换为mapjoin
set hive.auto.convert.join = true;
--小表的最大文件大小，默认为25000000，即25M
set hive.mapjoin.smalltable.filesize = 25000000;
--是否将多个mapjoin合并为一个
set hive.auto.convert.join.noconditionaltask = true;
--多个mapjoin转换为1个时，所有小表的文件大小总和的最大值。
set hive.auto.convert.join.noconditionaltask.size = 10000000;