解决：idea上传文件到hdfs上有文件无数据问题

阿里云的云服务器，使用内网搭建的集群：一下 默认为内网ip
整体环境：
xxx:xxx:xxx:xxx master
xxx:xxx:xxx:xxx slave1
xxx:xxx:xxx:xxx slave2
以master – 两个datanode的模式
首先说明：

• 防火墙已经关闭
• namenode，datanode都已经启动正常
• hadoop dfsadmin -report 查看各节点正常
• 50010端口在安全组里开放

namenode的clusterID

[hadoop@master current]$ cat VERSION 
#Fri May 22 10:33:28 CST 2020
namespaceID=1108140250
clusterID=CID-cc79e92a-7fa3-4036-95fc-0ced5ae8e539
cTime=1589249034335
storageType=NAME_NODE
blockpoolID=BP-2146019854-172.26.171.170-1589249034335
layoutVersion=-63

slave1的clusterID

[hadoop@slave1 current]$ cat VERSION 
#Fri May 22 10:33:32 CST 2020
storageID=DS-255a6196-1e22-4d3d-a564-9c28e45797b6
clusterID=CID-cc79e92a-7fa3-4036-95fc-0ced5ae8e539
cTime=0
datanodeUuid=4082ef56-f469-48c0-9cb8-b5790fbda6f2
storageType=DATA_NODE
layoutVersion=-57
[hadoop@slave1 current]$ ll

slave2的 clusterID

[hadoop@slave2 current]$ cat VERSION 
#Fri May 22 10:33:32 CST 2020
storageID=DS-6ec7370f-5c56-4117-9b6f-c040765aa3ab
clusterID=CID-cc79e92a-7fa3-4036-95fc-0ced5ae8e539
cTime=0
datanodeUuid=b2f8f1a2-bfec-4af4-9f79-1b58c1ab8f5f
storageType=DATA_NODE
layoutVersion=-57

在看启动进程

[hadoop@master current]$ jps
2642 SecondaryNameNode
2440 NameNode
3822 Jps
1519 ResourceManager
[hadoop@master current]$ 

[hadoop@slave1 /]$ jps
19523 NodeManager
19414 DataNode
19999 Jps
[hadoop@slave1 /]$ 

[hadoop@slave2 current]$ jps
7953 Jps
4641 DataNode
4750 NodeManager
[hadoop@slave2 current]$ 

防火墙也关闭了

上传hdfs时报错：

 WARN hdfs.DFSClient: DataStreamer Exception: org.apache.hadoop.ipc.RemoteException: java.io.IOException: File /k_test/hello.txt could only be replicated to 0 nodes, instead of 1
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1639)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:736)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)

解决办法：

把master、slave1、slave2的 公网ip配置在电脑的hosts中，做映射
C:\Windows\System32\drivers\etc
hosts文件中配置公网ip

182.92.xxx.xxx master  
39.97. xxx.xxx  slave1
39.97.xxx.xxx slave2

在代码中添加这个

config.set("dfs.client.use.datanode.hostname", "true");

完美解决上传