hdfs远程连接hadoop问题与解决

问题：使用hdfs客户端在本地连接阿里云服务器上部署的hadoop，操作hdfs时候出现异常：could only be replicated to 0 nodes instead of minReplication (=1). There are 1 datanode(s) running and 1 node(s) are excluded in this operation.

并且，在管理网页上查看文件文件大小全部为0；

原因：在百度找了很久都没有找答案都是说dataNode没启动，可是服务器上jps查看dataNode已经正常启动了，百思不得其解，后来在stackoverflow找到一段话：

It take me a week to figure out the problem in my situation.

When the client(your program) ask the nameNode for data operation, the nameNode picks up a dataNode and navigate the client to it, by giving the dataNode's ip to the client.

But, when the dataNode host is configured to has multiple ip, and the nameNode gives you the one your client CAN'T ACCESS TO, the client would add the dataNode to exclude list and ask the nameNode for a new one, and finally all dataNode are excluded, you get this error.

So check node's ip settings before you try everything!!!

大致的意思是：客户端操作hdfs时候先连接namenode，然后namenode分配给客户端一个datanoe的ip地址，如果这个ip地址客户端无法访问到，就会被客户端添加到排除列表中。

而我的阿里云服务器是多ip地址，所以分配给我了一个不可达到的地址；从而出现了这个问题；

解决：将客户端程序放在hadoop服务器上运行，该错误就会消失。在hadoop监管页面查看dataNode的节点是 iZ234nvolhdZ (10.253.102.93:50010)，我本地10.253.102.93地址可以ping通，而telnet端口号50010不通，说明是端口号的问题把50010端口号开通后本地就可以访问了。

参考：

https://stackoverflow.com/questions/5293446/hdfs-error-could-only-be-replicated-to-0-nodes-instead-of-1

http://blog.csdn.net/cht0112/article/details/72911307?utm_source=itdadao&utm_medium=referral

问题：ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: iZ234nvolhdZ:50010:DataXceiver error processing WRITE_BLOCK operation src: /10.253.102.93:59704 dst: /10.253.102.93:50010 java.io.IOException: Premature EOF from inputStream

原因：客户端上传文件流操作后忘记关掉

解决：输出流关闭和刷新

outputStream.flush();

outputStream.close();

参考： http://blog.csdn.net/menghuannvxia/article/details/44591619