记一次Client 端操作Hadoop出现的错误

                                 Error Log 哈哈哈哈

---

 很奇怪的一个问题,我之前操作都没有问题的,但是今天却给我遇见了，所以就贴出来做一些记录

 

  这是我想创建目录的代码

    @Test
    public void mkdirsTest() throws IOException, URISyntaxException, InterruptedException {

        // System.setProperty("hadoop.home.dir","D:\\Java_DaiMa\\java学习过程安装包\\hadoop\\hadoop-2.7.2");

        Configuration configuration = new Configuration();
        // configuration.set("fs.defaultFS","hdfs://192.168.119.140:9000");
        // FileSystem fileSystem = FileSystem.get(configuration);
        FileSystem fileSystem = FileSystem.get(new URI("hdfs://192.168.119.140:9000"), configuration, "root");
        fileSystem.mkdirs(new Path("/gavin_test/ga/gb"));
        fileSystem.close();
    }

2019-10-05 11:41:50,837 ERROR [org.apache.hadoop.util.Shell] - Failed to locate the winutils binary in the hadoop binary path
java.io.IOException: Could not locate executable null\bin\winutils.exe in the Hadoop binaries.
	at org.apache.hadoop.util.Shell.getQualifiedBinPath(Shell.java:356)
	at org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:371)
	at org.apache.hadoop.util.Shell.<clinit>(Shell.java:364)
	at org.apache.hadoop.util.StringUtils.<clinit>(StringUtils.java:80)
	at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:2807)
	at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:2802)
	at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2668)
	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:371)
	at org.apache.hadoop.fs.FileSystem$1.run(FileSystem.java:160)
	at org.apache.hadoop.fs.FileSystem$1.run(FileSystem.java:157)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:157)
	at com.yang.hdfs.YangHdsfClient.mkdirsTest(YangHdsfClient.java:28)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:305)
	at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:365)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
	at org.junit.runners.ParentRunner$4.run(ParentRunner.java:330)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:78)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:328)
	at org.junit.runners.ParentRunner.access$100(ParentRunner.java:65)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:292)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:305)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:412)
	at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
	at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
	at com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:47)
	at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242)
	at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)

这是错误的日志；

查看hadoop源码发现里有这么一段：

public static final String getQualifiedBinPath(String executable)

  throws IOException {

    // construct hadoop bin path to the specified executable

    String fullExeName = HADOOP_HOME_DIR + File.separator + "bin"

      + File.separator + executable;

 

    File exeFile = new File(fullExeName);

    if (!exeFile.exists()) {

      throw new IOException("Could not locate executable " + fullExeName

        + " in the Hadoop binaries.");

    }

 

    return exeFile.getCanonicalPath();

  }

 

private static String HADOOP_HOME_DIR = checkHadoopHome();

private static String checkHadoopHome() {

 

    // first check the Dflag hadoop.home.dir with JVM scope

    String home = System.getProperty("hadoop.home.dir");

 

    // fall back to the system/user-global env variable

    if (home == null) {

      home = System.getenv("HADOOP_HOME");

    }

     ...

}

很明显应该是HADOOP_HOME的问题。如果HADOOP_HOME为空，必然fullExeName为null\bin\winutils.exe。解决方法很简单，配置环境变量，不想重启电脑可以在程序里加上：

System.setProperty("hadoop.home.dir","D:\\Java_DaiMa\\java学习过程安装包\\hadoop\\hadoop-2.7.2");

D:\\Java_DaiMa\\java学习过程安装包\\hadoop\\hadoop-2.7.2  这个地址是我本地的地址.

稍后再执行，你可能还是会出现同样的错误，这个时候你可能会要怪我了。其实一开始我是拒绝的，因为你进入你的hadoop-x.x.x/bin目录下看，你会发现你压根就没有winutils.exe这个东东。

你可以去github下载一个 : https://github.com/srccodes/hadoop-common-2.2.0-bin  下载好后，把winutils.exe加入你的hadoop-x.x.x/bin下。

错误记录