Hive job,抛错could only be replicated to 0 nodes instead of minReplication(=1)

场景: hive执行一条sql语句,然后map,reduce做完了,写数据over了
hive> select dday.full_day, sum(quantity_ordered) from fact_order as fact inner join dim_day  as dday on fact.time_key == dday.day_key and dday.full_day >= "2015-05-01" and dday.full_day <= "2015-05-02" group by dday.full_day order by dday.full_day;
WARNING: Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. tez, spark) or using Hive 1.X releases.
Query ID = root_20160719160937_c80b5102-f24e-48f5-98a6-72f13d486805
Total jobs = 2
Execution log at: /tmp/root/root_20160719160937_c80b5102-f24e-48f5-98a6-72f13d486805.log
2016-07-19 16:11:19     Starting to launch local task to process map join;      maximum memory = 477102080
2016-07-19 16:11:23     Dump the side-table for tag: 1 with group count: 2 into file: file:/tmp/root/675fa67b-59c3-4de7-99a9-3f5be9f6685c/hive_2016-07-19_16-11-06_796_8611429774960723114-1/-local-10006/HashTable-Stage-2/MapJoin-mapfile01--.hashtable
2016-07-19 16:11:23     Uploaded 1 File to: file:/tmp/root/675fa67b-59c3-4de7-99a9-3f5be9f6685c/hive_2016-07-19_16-11-06_796_8611429774960723114-1/-local-10006/HashTable-Stage-2/MapJoin-mapfile01--.hashtable (340 bytes)
2016-07-19 16:11:23     End of local task; Time Taken: 3.306 sec.
Execution completed successfully
MapredLocal task succeeded
Launching Job 1 out of 2
Number of reduce tasks not specified. Estimated from input data size: 1
In order to change the average load for a reducer (in bytes):
  set hive.exec.reducers.bytes.per.reducer=
In order to limit the maximum number of reducers:
  set hive.exec.reducers.max=
In order to set a constant number of reducers:
  set mapreduce.job.reduces=
Starting Job = job_1468583637020_0001, Tracking URL = http://sht-sgmhadoopnn-02:8088/proxy/application_1468583637020_0001/
Kill Command = /hadoop/hadoop-2.7.2/bin/hadoop job  -kill job_1468583637020_0001
Hadoop job information for Stage-2: number of mappers: 1; number of reducers: 1
2016-07-19 16:11:38,839 Stage-2 map = 0%,  reduce = 0%
2016-07-19 16:11:47,502 Stage-2 map = 100%,  reduce = 0%, Cumulative CPU 2.59 sec
2016-07-19 16:11:55,025 Stage-2 map = 100%,  reduce = 100%, Cumulative CPU 4.41 sec
MapReduce Total cumulative CPU time: 4 seconds 410 msec
Ended Job = job_1468583637020_0001
Launching Job 2 out of 2
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
  set hive.exec.reducers.bytes.per.reducer=
In order to limit the maximum number of reducers:
  set hive.exec.reducers.max=
In order to set a constant number of reducers:
  set mapreduce.job.reduces=
java.lang.RuntimeException: Error caching map.xml: org.apache.hadoop.ipc.RemoteException(java.io.IOException): File /tmp/hive/root/675fa67b-59c3-4de7-99a9-3f5be9f6685c/hive_2016-07-19_16-11-06_796_8611429774960723114-1/-mr-10013/f1b1cc9d-19ef-448c-8b9d-3e48129c7fec/map.xml could only be replicated to 0 nodes instead of minReplication (=1). There are 3 datanode(s) running and no node(s) are excluded in this operation.

        at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:1547)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getNewBlockTargets(FSNamesystem.java:3107)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:3031)
        at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:724)
        at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:492)
        at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2049)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2045)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2043)

        at org.apache.hadoop.hive.ql.exec.Utilities.setBaseWork(Utilities.java:577)
        at org.apache.hadoop.hive.ql.exec.Utilities.setMapWork(Utilities.java:511)
        at org.apache.hadoop.hive.ql.exec.Utilities.setMapRedWork(Utilities.java:503)
        at org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:377)
        at org.apache.hadoop.hive.ql.exec.mr.MapRedTask.execute(MapRedTask.java:138)
        at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:158)
        at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:101)
        at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1840)
        at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1584)
        at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1361)
        at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1184)
        at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1172)
        at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:233)
        at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:184)
        at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:400)
        at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:778)
        at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:717)
        at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:645)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:136)

错误:
java.lang.RuntimeException: Error caching map.xml: org.apache.hadoop.ipc.RemoteException(java.io.IOException): File /tmp/hive/root/675fa67b-59c3-4de7-99a9-3f5be9f6685c
/hive_2016-07-19_16-11-06_796_8611429774960723114-1/-mr-10013/f1b1cc9d-19ef-448c-8b9d-3e48129c7fec/map.xml could only be replicated to 0 nodes instead of minReplication (=1).
There are 3 datanode(s) running and no node(s) are excluded in this operation.

分析:
1.检查集群服务状态是否正常
2.新建个txt文件,然后put到hdfs系统,再次检查
3.磁盘空间不足

 

解决方法:
a.检查各个节点的磁盘是否99%，100%
df -h

b.查询出大于100M的文件
find / -type f -size +100000000c -exec du -sh {} \;

c.删选不要的文件或者日志,删除,降低磁盘使用

验证:
hive> select dday.full_day, sum(quantity_ordered) from fact_order as fact inner join dim_day  as dday on fact.time_key == dday.day_key and dday.full_day >= "2015-05-01" and dday.full_day <= "2015-05-02" group by dday.full_day order by dday.full_day;
WARNING: Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. tez, spark) or using Hive 1.X releases.
Query ID = root_20160719160937_c80b5102-f24e-48f5-98a6-72f13d486805
Total jobs = 2
Execution log at: /tmp/root/root_20160719160937_c80b5102-f24e-48f5-98a6-72f13d486805.log
2016-07-19 18:35:26     Starting to launch local task to process map join;      maximum memory = 477102080
2016-07-19 18:35:30     Dump the side-table for tag: 1 with group count: 2 into file: file:/tmp/root/675fa67b-59c3-4de7-99a9-3f5be9f6685c/hive_2016-07-19_18-35-14_146_5682002849336110794-1/-local-10006/HashTable-Stage-2/MapJoin-mapfile11--.hashtable
2016-07-19 18:35:30     Uploaded 1 File to: file:/tmp/root/675fa67b-59c3-4de7-99a9-3f5be9f6685c/hive_2016-07-19_18-35-14_146_5682002849336110794-1/-local-10006/HashTable-Stage-2/MapJoin-mapfile11--.hashtable (340 bytes)
2016-07-19 18:35:30     End of local task; Time Taken: 3.977 sec.
Execution completed successfully
MapredLocal task succeeded
Launching Job 1 out of 2
Number of reduce tasks not specified. Estimated from input data size: 1
In order to change the average load for a reducer (in bytes):
  set hive.exec.reducers.bytes.per.reducer=
In order to limit the maximum number of reducers:
  set hive.exec.reducers.max=
In order to set a constant number of reducers:
  set mapreduce.job.reduces=
Starting Job = job_1468583637020_0002, Tracking URL = http://sht-sgmhadoopnn-02:8088/proxy/application_1468583637020_0002/
Kill Command = /hadoop/hadoop-2.7.2/bin/hadoop job  -kill job_1468583637020_0002
Hadoop job information for Stage-2: number of mappers: 1; number of reducers: 1
2016-07-19 18:35:42,929 Stage-2 map = 0%,  reduce = 0%
2016-07-19 18:35:51,468 Stage-2 map = 100%,  reduce = 0%, Cumulative CPU 2.25 sec
2016-07-19 18:35:58,951 Stage-2 map = 100%,  reduce = 100%, Cumulative CPU 4.49 sec
MapReduce Total cumulative CPU time: 4 seconds 490 msec
Ended Job = job_1468583637020_0002
Launching Job 2 out of 2
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
  set hive.exec.reducers.bytes.per.reducer=
In order to limit the maximum number of reducers:
  set hive.exec.reducers.max=
In order to set a constant number of reducers:
  set mapreduce.job.reduces=
Starting Job = job_1468583637020_0003, Tracking URL = http://sht-sgmhadoopnn-02:8088/proxy/application_1468583637020_0003/
Kill Command = /hadoop/hadoop-2.7.2/bin/hadoop job  -kill job_1468583637020_0003
Hadoop job information for Stage-3: number of mappers: 1; number of reducers: 1
2016-07-19 18:36:08,812 Stage-3 map = 0%,  reduce = 0%
2016-07-19 18:36:16,285 Stage-3 map = 100%,  reduce = 0%, Cumulative CPU 1.39 sec
2016-07-19 18:36:23,767 Stage-3 map = 100%,  reduce = 100%, Cumulative CPU 3.41 sec
MapReduce Total cumulative CPU time: 3 seconds 410 msec
Ended Job = job_1468583637020_0003
MapReduce Jobs Launched:
Stage-Stage-2: Map: 1  Reduce: 1   Cumulative CPU: 4.49 sec   HDFS Read: 13812 HDFS Write: 158 SUCCESS
Stage-Stage-3: Map: 1  Reduce: 1   Cumulative CPU: 3.41 sec   HDFS Read: 5406 HDFS Write: 30 SUCCESS
Total MapReduce CPU Time Spent: 7 seconds 900 msec
OK
2015-05-01      600
2015-05-02      600
Time taken: 70.768 seconds, Fetched: 2 row(s)
hive>

来自 “ ITPUB博客 ” ，链接：http://blog.itpub.net/30089851/viewspace-2126303/，如需转载，请注明出处，否则将追究法律责任。

转载于:http://blog.itpub.net/30089851/viewspace-2126303/