hadoop 集群遇到的问题汇总

有时候报错会误导问题的定位和排查，比如下面这一款

org.apache.hadoop.ipc.RemoteException(java.io.IOException): File /user/root/.sparkStaging/application_1539063128463_0573/__spark_libs__2844575839698714585.zip could only be replicated to 0 nodes instead of minReplication (=1).  There are 5 datanode(s) running and no node(s) are excluded in this operation.
	at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:1571)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getNewBlockTargets(FSNamesystem.java:3107)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:3031)
	at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:725)
	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:492)
	at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2049)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2045)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2043)

刚开始时以为是集群问题，反复重启也没有解决。
其实是hdfs 存储过满，清掉一部分存储即可。

下面的问题很古怪，一般人肯定想不到

Note: System times on machines may be out of sync. Check system time and time zones.
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.instantiateException(SerializedExceptionPBImpl.java:168)
        at org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.deSerialize(SerializedExceptionPBImpl.java:106)
        at org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher.launch(AMLauncher.java:122)
        at org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher.run(AMLauncher.java:250)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
. Failing the application.
         ApplicationMaster host: N/A
         ApplicationMaster RPC port: -1
         queue: root.root
         start time: 1541957975386 
         final status: FAILED
         tracking URL: http://bdpe101:8088/cluster/app/application_1539063128463_0745
         user: root
18/11/12 01:39:36 INFO Client: Deleted staging directory hdfs://bdpe101:9000/user/root/.sparkStaging/application_1539063128463_0745
18/11/12 01:39:36 ERROR SparkContext: Error initializing SparkContext.
org.apache.spark.SparkException: Yarn application has already ended! It might have been killed or unable to launch application master.

万万想不到，竟然是集群时钟不同步的问题导致的

Production FAQ

参考链接：

• https://blog.csdn.net/s646575997/article/details/51290394

java.net.SocketTimeoutException: 480000 millis timeout while waiting for channel to be ready for write

原因：IO超时

解决方法：
修改hadoop配置文件hdfs-site.xml，增加dfs.datanode.socket.write.timeout和dfs.socket.timeout两个属性的设置。

<property>
    <name>dfs.datanode.socket.write.timeout</name>
    <value>6000000</value>
</property>

<property>
        <name>dfs.socket.timeout</name>
        <value>6000000</value>
</property>

DataXceiver error processing WRITE_BLOCK operation

原因：文件操作超租期，实际上就是data stream操作过程中文件被删掉了。
解决办法：
修改hdfs-site.xml （针对2.x版本，1.x版本属性名应该是：dfs.datanode.max.xcievers）：

<property> 
        <name>dfs.datanode.max.transfer.threads</name>
        <value>40960</value> 
</property>

java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try.

原因：无法写入；假设环境中有3个datanode，备份数量设置的是3。在写操作时，它会在pipeline中写3个机器。默认replace-datanode-on-failure.policy是DEFAULT,如果系统中的datanode大于等于3，它会找另外一个datanode来拷贝。目前机器只有3台，因此只要一台datanode出问题，就一直无法写入成功。)

<property>
    <name>dfs.client.block.write.replace-datanode-on-failure.enable</name>
    <value>true</value>
</property>
<property>
    <name>dfs.client.block.write.replace-datanode-on-failure.policy</name>
    <value>NEVER</value>
</property>

或者

<property>
    <name>dfs.client.block.write.replace-datanode-on-failure.enable</name>
    <value>false</value>
</property>