hadoop错误问题总结

1.namenode循环报错如下：
2012-08-21 09:20:24,486 WARN org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Cannot roll edit log, edits.new files already exists in all healthy directories:
/data/work/hdfs/name/current/edits.new
/backup/current/edits.new
2012-08-21 09:20:25,357 ERROR org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:hadoop cause:java.net.ConnectException: Connection refused
2012-08-21 09:20:25,357 ERROR org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:hadoop cause:java.net.ConnectException: Connection refused
2012-08-21 09:20:25,359 WARN org.mortbay.log: /getimage: java.io.IOException: GetImage failed. java.net.ConnectException: Connection refused

secondarynamenode也有相关错误。
搜到一个说法原因是：
With 1.0.2, only one checkpoint process is executed at a time. When the namenode gets an overlapping checkpointing request, it checks edit.new in its storage directories. If all of them have this file, namenode concludes the previous checkpoint process is not done yet and prints the warning message you've seen.

这样的话如果你确保edits.new文件是之前错误操作残留下的没有用的文件的话，那么可以删掉，检测之后是否还有这样的问题。
另外请确保namenode的hdfs-site.xml的配置有如下项：
<property>
<name>dfs.secondary.http.address</name>
<value>0.0.0.0:50090</value>
</property>

将上述的0.0.0.0修改为你部署secondarynamenode的主机名
secondarynamenode的hdfs-site.xml有如下项：
<property>
<name>dfs.http.address</name>
<value>0.0.0.0:50070</value>
</property>

将上述的0.0.0.0修改为你部署namenode的主机名


2.namenode循环报错如下：
2014-06-18 09:28:12,280 ERROR org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:hive1 cause:java.io.IOException: File /tmp/hive-hive1/hive_2014-06-18_09-28-11_245_4808015493735494213-1/-mr-10003/b47c8250-3fea-4b5f-b015-3a62cf97ce53/reduce.xml could only be replicated to 0 nodes, instead of 1
2014-06-18 09:28:12,280 INFO org.apache.hadoop.ipc.Server: IPC Server handler 33 on 54310, call addBlock(/tmp/hive-hive1/hive_2014-06-18_09-28-11_245_4808015493735494213-1/-mr-10003/b47c8250-3fea-4b5f-b015-3a62cf97ce53/reduce.xml, DFSClient_NONMAPREDUCE_-430436880_1, null) from 192.168.49.94:45419: error: java.io.IOException: File /tmp/hive-hive1/hive_2014-06-18_09-28-11_245_4808015493735494213-1/-mr-10003/b47c8250-3fea-4b5f-b015-3a62cf97ce53/reduce.xml could only be replicated to 0 nodes, instead of 1
java.io.IOException: File /tmp/hive-hive1/hive_2014-06-18_09-28-11_245_4808015493735494213-1/-mr-10003/b47c8250-3fea-4b5f-b015-3a62cf97ce53/reduce.xml could only be replicated to 0 nodes, instead of 1
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1920)
at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:783)
at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:587)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1432)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1428)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1426)
2014-06-18 09:28:12,286 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Increasing replication for /tmp/hive-hive1/hive_2014-06-18_09-28-11_245_4808015493735494213-1/-mr-10003/b47c8250-3fea-4b5f-b015-3a62cf97ce53/reduce.xml. New replication is 10
2014-06-18 09:28:12,456 WARN org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Not able to place enough replicas, still in need of 3 to reach 3
Not able to place enough replicas


同时查看jobtracker的日志，也是循环抛出异常，日志如下：
2014-06-18 09:49:20,725 WARN org.apache.hadoop.hdfs.DFSClient: DataStreamer Exception: org.apache.hadoop.ipc.RemoteException: java.io.IOEx ception: File /home/sa/hadoop-1.2.1/mapred/sys/jobtracker.info could only be replicated to 0 nodes, instead of 1
9916 at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1920)
9917 at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:783)
9918 at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)
9919 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
9920 at java.lang.reflect.Method.invoke(Method.java:606)
9921 at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:587)
9922 at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1432)
9923 at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1428)
9924 at java.security.AccessController.doPrivileged(Native Method)
9925 at javax.security.auth.Subject.doAs(Subject.java:415)
9926 at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
9927 at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1426)
9928
9929 at org.apache.hadoop.ipc.Client.call(Client.java:1113)
9930 at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229)
9931 at com.sun.proxy.$Proxy7.addBlock(Unknown Source)
9932 at sun.reflect.GeneratedMethodAccessor9.invoke(Unknown Source)
9933 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
9934 at java.lang.reflect.Method.invoke(Method.java:606)
9935 at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:85)
9936 at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:62)
9937 at com.sun.proxy.$Proxy7.addBlock(Unknown Source)
9938 at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:3720)
9939 at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:3580)
9940 at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2600(DFSClient.java:2783)
9941 at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:3023)

解决方案：网上搜了的方案都是了，结果还是不行。仔细看日志Not able to place enough replicas，于是查看节点的空间，发现都还有一些空间。怀疑是配置文件block设置过大的原因，于是查看hadoop的配置文件，配置如下：
<property>
<name>dfs.block.size</name>
<value>268435456</value>
</property>
<property>
block块大小是256M也不大，应该不会出现空间不足的问题。另外还有一个配置
<property>
<name>dfs.datanode.du.reserved</name>
<value>10474836480</value>
</property>
配置磁盘预留空间比较大，于是干掉。另外为了节省空间将dfs.replication设置为1.重启集群错误问题消失。