在一次用maven-assembly打包HDFS项目成jar后运行出现了以下错误(maven-assembly可以将Maven依赖的Jar都一起打包进来)
log4j:WARN No appenders could be found for logger (org.apache.hadoop.metrics2.lib.MutableMetricsFactory).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
Exception in thread "main" java.io.IOException: No FileSystem for scheme: hdfs
at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2421)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2428)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:88)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2467)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2449)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:367)
at org.apache.hadoop.fs.FileSystem$1.run(FileSystem.java:156)
at org.apache.hadoop.fs.FileSystem$1.run(FileSystem.java:153)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:153)
at com.cetc.di.HDFSFileSystem.<init>(HDFSFileSystem.java:41)
at callhdfs.Main.main(Main.java:11)
在一番排查和查阅资料后发现使用Maven-assembly-plugin(fatjar也是一样的)的时候会覆盖相同的文件,然而HDFS的hadoop-hdfs-2.7.1.jar和hadoop-common-2.7.1.jar都有org.apache.hadoop.hdfs.DistributedFileSystem,将会进行覆盖。在运行的过程中使用了hdfs://IP : port的schema,而在生成的最终jar包中,无法找到这个schema的实现。所以就抛出了
解决方案
在代码中加入:
configuration.set("fs.hdfs.impl", "org.apache.hadoop.hdfs.DistributedFileSystem");