测试客户端往Hadoop节点中上传和下载数据过程中遇到的问题

本文链接：https://blog.csdn.net/Hodors/article/details/98482492

前提：

hadoop节点是已经配置好的：

能够在浏览器中打开

例如：

192.168.8.888:9000

需要注意的事项

客户端的端口号要和服务器的端口号保持一致！

然后

上传数据的code

try {
	Configuration conf;
	FileSystem hdfs;
	conf = new Configuration();
	conf.set("fs.default.name","hdfs://192.168.8.888:9000");
	hdfs = FileSystem.get(conf);
	Path src = new Path("D:\\JavaProject\\hjgljava\\hjglTestData\\888.zip");
	Path dst = new Path("/user/hadoop/alldata/origindata");
	System.out.println("Uploading....");
	hdfs.copyFromLocalFile(src,dst);
	System.out.println("Uploading "+ dataID+".zip" + " Done!");
	fishedFlag = true;
	return 1;
}catch(Exception eee){
	eee.printStackTrace();
	return -1;
}

下载数据的code

try {
	Configuration conf = new Configuration();
	conf.set("fs.default.name","hdfs://192.168.8.888:9000");
	FileSystem hdfs = FileSystem.get(conf);																	
	String TmpFilePath = "D:\\JavaProject\\hjgljava\\hjglTestData\\DownloadData\\";
	String DataName = "888"+".zip";
	Path srcdown = new Path("/user/hadoop/alldata/origindata/"+DataName);
	Path down = new Path(TmpFilePath);
	System.out.println("Downloading..."+srcdown);
	hdfs.copyToLocalFile(false, srcdown, down, true);
	System.out.println("Downloading..."+DataName+"...Done!");
	return 1;
} catch (IOException e) {
	e.printStackTrace();
	return -1;
}

配置过程终于到的问题

hdfs://192.168.8.888:50070
/user/hadoop/alldata/origindata
2019-08-05 10:22:51,405 INFO  [Thread-0] Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1243)) - fs.default.name is deprecated. Instead, use fs.defaultFS
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.apache.hadoop.security.authentication.util.KerberosUtil (file:/D:/hadoop/hadoop276/share/hadoop/common/lib/hadoop-auth-2.7.6.jar) to method sun.security.krb5.Config.getInstance()
WARNING: Please consider reporting this to the maintainers of org.apache.hadoop.security.authentication.util.KerberosUtil
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Uploading....
java.io.IOException: Failed on local exception: com.google.protobuf.InvalidProtocolBufferException: Protocol message end-group tag did not match expected tag.; Host Details : local host is: "FengZhihengLaptop/192.168.0.126"; destination host is: "192.168.0.12":50070; 
	at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:776)
	at org.apache.hadoop.ipc.Client.call(Client.java:1480)
	at org.apache.hadoop.ipc.Client.call(Client.java:1413)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
	at com.sun.proxy.$Proxy13.getFileInfo(Unknown Source)
	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:776)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191)
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
	at com.sun.proxy.$Proxy14.getFileInfo(Unknown Source)
	at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:2117)
	at org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1305)
	at org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1301)
	at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
	at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1317)
	at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1425)
	at org.apache.hadoop.fs.FileUtil.checkDest(FileUtil.java:496)
	at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:348)
	at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:338)
	at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1968)
	at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1936)
	at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1901)
	at com.hjgl.uploadFrame.uploadOneTestGroupData(uploadFrame.java:231)
	at com.hjgl.Main$5$6.run(Main.java:965)
Caused by: com.google.protobuf.InvalidProtocolBufferException: Protocol message end-group tag did not match expected tag.
	at com.google.protobuf.InvalidProtocolBufferException.invalidEndTag(InvalidProtocolBufferException.java:94)
	at com.google.protobuf.CodedInputStream.checkLastTagWas(CodedInputStream.java:124)
	at com.google.protobuf.AbstractParser.parsePartialFrom(AbstractParser.java:202)
	at com.google.protobuf.AbstractParser.parsePartialDelimitedFrom(AbstractParser.java:241)
	at com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:253)
	at com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:259)
	at com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:49)
	at org.apache.hadoop.ipc.protobuf.RpcHeaderProtos$RpcResponseHeaderProto.parseDelimitedFrom(RpcHeaderProtos.java:3167)
	at org.apache.hadoop.ipc.Client$Connection.receiveRpcResponse(Client.java:1087)
	at org.apache.hadoop.ipc.Client$Connection.run(Client.java:980)