Hadoop: File could only be replicated to 0 nodes instead of minReplication (=1). There are 1 datanod

Issue:

When you upload a file to HDFS, you may encounter following exception.

Exception:

Exception in thread "main" org.apache.hadoop.ipc.RemoteException(java.io.IOException): File /test222.xml could only be replicated to 0 nodes instead of minReplication (=1).  There are 1 datanode(s) running and 1 node(s) are excluded in this operation.

Reason:

If you are sure the capacity of data node is not exhausted and the communication between master node and data node is working well, then the issue may be caused by url resolving. Specifically, HDFS client cannot connect to datanode, although it can be connected with master node.

Solution:

1. add following configuration on the org.apache.hadoop.conf.Configuration instance:

Configuration conf = new Configuration();
conf.set("dfs.client.use.datanode.hostname", "true");

2. confirm your client could ping datanode

E.g. datanode host name: quickstart.cloudera

If ping is failed, then add resolving mapping manually.

1) In Mac OS, use following command to edit hosts file.

sudo vi /private/etc/hosts

2) add following mapping to the file.

127.0.0.1       quickstart.cloudera

Then, try your program again. The issue should be gone.