java 解决Could not locate executable null\bin\winutils.exe in the Hadoop bin

问题描述
在windows环境下运行连接hadoop服务的程序报告以下错误信息：

og4j:ERROR Could not find value for key log4j.appender.logRollingFile
log4j:ERROR Could not instantiate appender named "logRollingFile".
[ERROR] 2018-09-17 17:40:02,151 method:org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:374)
Failed to locate the winutils binary in the hadoop binary path
java.io.IOException: Could not locate executable null\bin\winutils.exe in the Hadoop binaries.
    at org.apache.hadoop.util.Shell.getQualifiedBinPath(Shell.java:356)
    at org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:371)
    at org.apache.hadoop.util.Shell.<clinit>(Shell.java:364)
    at org.apache.hadoop.util.StringUtils.<clinit>(StringUtils.java:80)
    at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:2807)
    at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:2802)
    at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2668)
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:371)
    at org.apache.hadoop.fs.FileSystem$1.run(FileSystem.java:160)
    at org.apache.hadoop.fs.FileSystem$1.run(FileSystem.java:157)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:157)
    at com.yue.modular.hdfsClient.main(hdfsClient.java:31)
[WARN ] 2018-09-17 17:40:03,432 method:org.apache.hadoop.util.NativeCodeLoader.<clinit>(NativeCodeLoader.java:62)
Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

原因分析
根据错误信息进入hadoop源码发现：

static {
        WINDOWS = osType == Shell.OSType.OS_TYPE_WIN;
        SOLARIS = osType == Shell.OSType.OS_TYPE_SOLARIS;
        MAC = osType == Shell.OSType.OS_TYPE_MAC;
        FREEBSD = osType == Shell.OSType.OS_TYPE_FREEBSD;
        LINUX = osType == Shell.OSType.OS_TYPE_LINUX;
        OTHER = osType == Shell.OSType.OS_TYPE_OTHER;
        PPC_64 = System.getProperties().getProperty("os.arch").contains("ppc64");

        //注意这行
        HADOOP_HOME_DIR = checkHadoopHome();
        WINUTILS = getWinUtilsPath();
        isSetsidAvailable = isSetsidSupported();
        TOKEN_SEPARATOR_REGEX = WINDOWS ? "[|\n\r]" : "[ \t\n\r\f]";
    }

private static String checkHadoopHome() {

        //高能高能~~~~~~~~~
        String home = System.getProperty("hadoop.home.dir");
        if (home == null) {
            home = System.getenv("HADOOP_HOME");
        }

        try {
            if (home == null) {
                throw new IOException("HADOOP_HOME or hadoop.home.dir are not set.");
            }

            if (home.startsWith("\"") && home.endsWith("\"")) {
                home = home.substring(1, home.length() - 1);
            }

            File homedir = new File(home);
            if (!homedir.isAbsolute() || !homedir.exists() || !homedir.isDirectory()) {
                throw new IOException("Hadoop home directory " + homedir + " does not exist, is not a directory, or is not an absolute path.");
            }

            home = homedir.getCanonicalPath();
        } catch (IOException var2) {
            if (LOG.isDebugEnabled()) {
                LOG.debug("Failed to detect a valid hadoop home directory", var2);
            }

            home = null;
        }

        return home;
    }

public static final String getQualifiedBinPath(String executable) throws IOException {
        String fullExeName = HADOOP_HOME_DIR + File.separator + "bin" + File.separator + executable;
        File exeFile = new File(fullExeName);
        if (!exeFile.exists()) {
            throw new IOException("Could not locate executable " + fullExeName + " in the Hadoop binaries.");
        } else {
            return exeFile.getCanonicalPath();
        }
    }

问题解决
分析发现是hadoop.home.dir 为null 和获取HADOOP_HOME 变量没找到目录没找到，
解决办法2种：
第一种方法：添加环境变量HADOOP_HOME
第二种方法：直接java设置hadoop.home.dir 指向 winutil.exe 的目录
主要说一下设置hadoop.home.dir 目录指向：

        System.setProperty("hadoop.home.dir", "C:\\Program Files\\Hadoop\\hadoop2.7.2-bin");

        //构造一个访问指定hdfs系统的客户端对象
        FileSystem fs = FileSystem.get(URI.create("hdfs://192.168.119.129:9000"), new Configuration(), "root");

winutils.exe下载地址：
winutils.exe下载地址