今天打算从hive-0.14.0中剥离出一个miniHS2,但是在剥离的过程中抛出了一个NullPointerException,记一下排查问题的过程
下面是报错,几年前在写Mapreduce的时候,喷到过**Could not locate executable null\bin\winutils.exe in the Hadoop binaries.**这个错误,只需要配置一个winutils就可以了,但是想了一下是不是可以忽略这个报错呢,就没去管,最后发现还是因为winutils.exe引起的,不过也整好了解了一下为什么需要winutils。
java.io.IOException: Could not locate executable null\bin\winutils.exe in the Hadoop binaries.
at org.apache.hadoop.util.Shell.getQualifiedBinPath(Shell.java:355)
at org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:370)
at org.apache.hadoop.util.Shell.<clinit>(Shell.java:363)
at org.apache.hadoop.hive.conf.HiveConf$ConfVars.findHadoopBinary(HiveConf.java:2065)
at org.apache.hadoop.hive.conf.HiveConf$ConfVars.<clinit>(HiveConf.java:332)
at org.apache.hadoop.hive.conf.HiveConf.<clinit>(HiveConf.java:95)
at org.apache.hive.jdbc.LevinTest.main(LevinTest.java:33)
2019-07-12 10:38:48,505 WARN [main] util.NativeCodeLoader (NativeCodeLoader.java:<clinit>(62)) - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Exception in thread "main" java.lang.NullPointerException
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1012)
at org.apache.hadoop.util.Shell.runCommand(Shell.java:482)
at org.apache.hadoop.util.Shell.run(Shell.java:455)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702)
at org.apache.hadoop.util.Shell.execCommand(Shell.java:791)
at org.apache.hadoop.util.Shell.execCommand(Shell.java:774)
at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:646)
at org.apache.hadoop.fs.FilterFileSystem.setPermission(FilterFileSystem.java:472)
at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:597)
at org.apache.hive.jdbc.miniHS2.MiniHS2.<init>(MiniHS2.java:192)
at org.apache.hive.jdbc.miniHS2.MiniHS2.<init>(MiniHS2.java:220)
at org.apache.hive.jdbc.miniHS2.MiniHS2.<init>(MiniHS2.java:216)
at org.apache.hive.jdbc.LevinTest.main(LevinTest.java:35)
Disconnected from the target VM, address: '127.0.0.1:63904', transport: 'socket'
看了一下方法执行栈,是从下面的类开始的
MiniHS2.java-> FileSystem.mkdirs(fs, wareHouseDir, FULL_PERM); //创建warehouse路径,并且授权
FileSystem.java -> // 问题出在了fs.setPermission(dir, permission); 中
public static boolean mkdirs(FileSystem fs, Path dir, FsPermission permission)
throws IOException {
// create the directory using the default permission
boolean result = fs.mkdirs(dir);
// set its permission to be the supplied one
fs.setPermission(dir, permission);
return result;
}
RawLocalFileSystem.java -> // Shell.getSetPermissionCommand()方法会返回一个数组,问题就在这个
public void setPermission(Path p, FsPermission permission)
throws IOException {
if (NativeIO.isAvailable()) {
NativeIO.POSIX.chmod(pathToFile(p).getCanonicalPath(),
permission.toShort());
} else {
String perm = String.format("%04o", permission.toShort());
Shell.execCommand(Shell.getSetPermissionCommand(perm, false,
FileUtil.makeShellPath(pathToFile(p), true)));
}
}
Shell.java -> //继续跟到getSetPermissionCommand()中,马上就见到真像了
public static String[] getSetPermissionCommand(String perm, boolean recursive,
String file) {
String[] baseCmd = getSetPermissionCommand(perm, recursive);
String[] cmdWithFile = Arrays.copyOf(baseCmd, baseCmd.length + 1);
cmdWithFile[cmdWithFile.length - 1] = file;
return cmdWithFile;
}
Shell.java -> //下面代码很简单,如果是WINDOWS系统,在返回这个命令数组的时候会在[0]拼接一个WINUTILS路径,
但是我的环境下执行这个数组的结构变成了这样 new String {null,"chmod","-R","777"},有个可能还不是很清楚,继续往下走
public static String[] getSetPermissionCommand(String perm, boolean recursive) {
if (recursive) {
return (WINDOWS) ? new String[] { WINUTILS, "chmod", "-R", perm }
: new String[] { "chmod", "-R", perm };
} else {
return (WINDOWS) ? new String[] { WINUTILS, "chmod", perm }
: new String[] { "chmod", perm };
}
}
ProcessBuilder.java -> 看到下面的代码一下就明白了,在之前的debug的时候发现数组结构是这样的
[1] chmod
[2] -R
[3] 777
因为[0]是null,所以intellij就自动给忽略了,我也没有注意到
public Process start() throws IOException {
...
for (String arg : cmdarray)
if (arg == null)
throw new NullPointerException();
// Throws IndexOutOfBoundsException if command is empty
String prog = cmdarray[0];
...
}