论jar包的重要性

作为Java的小白，在jar包依赖的踩坑之路

第一坑 jar包冲突

背景：
kafka avro 初始化报错，找不到某类
原因：
hadoop-common中的vcore包和kafka的包冲突
解决
移除hadoop-common中的vcore包依赖

第二坑 jar包引用和客户端冲突

背景
Java初始化客户端，使用filesystem.get与HDFS服务端连接时，报以下错误：

+ java -cp HDFSPathMonitor-1.0-SNAPSHOT-jar-with-dependencies.jar com.server.HdfsPathMonitor
2021-03-08 16:51:29  INFO [Client.java:707] org.apache.hadoop.ipc.Client.handleConnectionFailure() - Retrying connect to server: ns1/****，:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
2021-03-08 16:51:30  INFO [Client.java:707] org.apache.hadoop.ipc.Client.handleConnectionFailure() - Retrying connect to server: ns1/10.17.1.252:8020. Already tried 1 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
2021-03-08 16:51:31  INFO [Client.java:707] org.apache.hadoop.ipc.Client.handleConnectionFailure() - Retrying connect to server: ns1/10.17.1.252:8020. Already tried 2 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
2021-03-08 16:51:32  INFO [Client.java:707] org.apache.hadoop.ipc.Client.handleConnectionFailure() - Retrying connect to server: ns1/10.17.1.252:8020. Already tried 3 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
2021-03-08 16:51:33  INFO [Client.java:707] org.apache.hadoop.ipc.Client.handleConnectionFailure() - Retrying connect to server: ns1/10.17.1.252:8020. Already tried 4 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
2021-03-08 16:51:34  INFO [Client.java:707] org.apache.hadoop.ipc.Client.handleConnectionFailure() - Retrying connect to server: ns1/10.17.1.252:8020. Already tried 5 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
2021-03-08 16:51:35  INFO [Client.java:707] org.apache.hadoop.ipc.Client.handleConnectionFailure() - Retrying connect to server: ns1/10.17.1.252:8020. Already tried 6 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
2021-03-08 16:51:36  INFO [Client.java:707] org.apache.hadoop.ipc.Client.handleConnectionFailure() - Retrying connect to server: ns1/10.17.1.252:8020. Already tried 7 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
2021-03-08 16:51:37  INFO [Client.java:707] org.apache.hadoop.ipc.Client.handleConnectionFailure() - Retrying connect to server: ns1/10.17.1.252:8020. Already tried 8 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
2021-03-08 16:51:38  INFO [Client.java:707] org.apache.hadoop.ipc.Client.handleConnectionFailure() - Retrying connect to server: ns1/10.17.1.252:8020. Already tried 9 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
2021-03-08 16:51:38 ERROR [UserGroupInformation.java:1193] org.apache.hadoop.security.UserGroupInformation.doAs() - PriviledgedActionException as:hadoop cause:java.net.ConnectException: Call to ns1/10.17.1.252:8020 failed on connection exception: java.net.ConnectException: 拒绝连接
java.net.ConnectException: Call to ns1/10.17.1.252:8020 failed on connection exception: java.net.ConnectException: 拒绝连接
	at org.apache.hadoop.ipc.Client.wrapException(Client.java:1142)
	at org.apache.hadoop.ipc.Client.call(Client.java:1118)
	at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229)
	at com.sun.proxy.$Proxy1.getProtocolVersion(Unknown Source)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:85)
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:62)
	at com.sun.proxy.$Proxy1.getProtocolVersion(Unknown Source)
	at org.apache.hadoop.ipc.RPC.checkVersion(RPC.java:422)
	at org.apache.hadoop.hdfs.DFSClient.createNamenode(DFSClient.java:183)
	at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:281)
	at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:245)
	at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:100)
	at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1446)
	at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:67)
	at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1464)
	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:263)
	at org.apache.hadoop.fs.FileSystem$1.run(FileSystem.java:118)
	at org.apache.hadoop.fs.FileSystem$1.run(FileSystem.java:116)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:116)
	at com.utils.HdfsUtil.init(HdfsUtil.java:38)
	at com.utils.HdfsUtil.<init>(HdfsUtil.java:29)
	at com.utils.Ks3Util.getCountHOfPath(Ks3Util.java:274)
	at com.server.HdfsPathMonitor.main(HdfsPathMonitor.java:99)
Caused by: java.net.ConnectException: 拒绝连接
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
	at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
	at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:511)
	at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:481)
	at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:457)
	at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:583)
	at org.apache.hadoop.ipc.Client$Connection.access$2200(Client.java:205)
	at org.apache.hadoop.ipc.Client.getConnection(Client.java:1249)
	at org.apache.hadoop.ipc.Client.call(Client.java:1093)
	... 28 more
Exception in thread "main" java.lang.NullPointerException
	at com.utils.HdfsUtil.getSubDirList(HdfsUtil.java:98)
	at com.utils.Ks3Util.getCountHOfPath(Ks3Util.java:276)
	at com.server.HdfsPathMonitor.main(HdfsPathMonitor.java:99)

代码和conf配置，都没有发现问题
conf

fs.defaultFS: hdfs://ns1
dfs.nameservices: ns1
dfs.ha.namenodes.ns1: nn1,nn2
dfs.namenode.rpc-address.ns1.nn1: hdp-01.dns.com:8020
dfs.namenode.rpc-address.ns1.nn2: hdp-02.dns.com:8020
dfs.client.failover.proxy.provider.ns1: org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider
fs.hdfs.impl: org.apache.hadoop.hdfs.DistributedFileSystem
hdfs.user: hadoop

代码

Configuration conf = new Configuration();
        Properties props = Constants.getHdfsProperties();
        for (String key : props.stringPropertyNames()) {
            conf.set(key, props.getProperty(key));
            log.info("加载HDFS配置文件如下：");
            log.info("{} = {}", key, props.getProperty(key));
        }
        String uri= props.getProperty("fs.defaultFS");
        String user=props.getProperty("hdfs.user");
        FileSystem fs =null;
        try {
            fs = FileSystem.get(new URI(uri), conf, user);
        } catch (InterruptedException e) {
            log.warn("init Interrupted! {}", e);
            Thread.currentThread().interrupt();
        } catch (IOException e) {
            e.printStackTrace();
        } catch (URISyntaxException e) {
            e.printStackTrace();
        }
        return fs;

修改fs.defaultFS为hdp-01.dns.com:8020后，报
Server IPC version 9 cannot communicate with client version 4
使用idea的maven项目运行mapreduce程序Server IPC version 9 cannot communicate with client version 4

原因：

Java初始化hdfs client时出现的，原因是maven依赖的版本和hdfs的版本不一致

解决方法
maven仓库中的hadoop-core最高只有1.2.1版本， 不能加这个配置， 使用hadoop-common + hadoop-hdfs + hadoop-client的2.7.1版本即可