在hive的源码中经常可以看到shims相关的类,shims相关类是用来兼容不同的hadoop和hive版本的,以HadoopShims为例org.apache.hadoop.hive.shims.HadoopShims是一个接口,具体的实现类为
org.apache.hadoop.hive.shims.Hadoop20Shims
org.apache.hadoop.hive.shims.Hadoop20SShims
org.apache.hadoop.hive.shims.Hadoop23Shims
具体对应的那个shims类在ShimLoader类(org.apache.hadoop.hive.shims.ShimLoader)中有所定义,以HADOOP_SHIM_CLASSES 为例:
/**
* The names of the classes for shimming Hadoop for each major version.
*/
private static final HashMap<String, String> HADOOP_SHIM_CLASSES =
new HashMap<String, String>(); // 首先定义了一个hashmap
static {
HADOOP_SHIM_CLASSES.put( "0.20", "org.apache.hadoop.hive.shims.Hadoop20Shims" ); // 0.20
HADOOP_SHIM_CLASSES.put( "0.20S", "org.apache.hadoop.hive.shims.Hadoop20SShims" ); // 1.x和0.20 security版本的hadoop
HADOOP_SHIM_CLASSES.put( "0.23", "org.apache.hadoop.hive.shims.Hadoop23Shims" ); // 2.x版本的hadoop
} // 向hashmap中插入对应的项,key为version信息,value为实际对应的类
ShimLoader类通过getHadoopShims方法返回具体的HadoopShims
private static HadoopShims hadoopShims;
......
public static synchronized HadoopShims getHadoopShims() {
if (hadoopShims == null) {
hadoopShims = loadShims(HADOOP_SHIM_CLASSES, HadoopShims.class); // 调用loadShims方法
}
return hadoopShims;
}
而loadShims方法通过判断hadoop的版本,来获取hashmap中对应的设置类
private static <T> T loadShims(Map<String, String> classMap, Class<T> xface) {
String vers = getMajorVersion(); // 获取hadoop的版本
String className = classMap.get(vers);
return createShim(className, xface);
}
具体哪个hadoop版本是由getMajorVersion获取,比如我们线上使用的是2.0.0-cdh4.6.0,那么getMajorVersion返回的值为0.23,对应的HadoopShims 实现类为org.apache.hadoop.hive.shims.Hadoop23Shims
public static String getMajorVersion() {
String vers = VersionInfo.getVersion(); // org.apache.hadoop.util.VersionInfo类的getVersion方法 比如2.0.0-cdh4.6.0
String[] parts = vers.split( "\\.");
if (parts. length < 2) {
throw new RuntimeException("Illegal Hadoop Version: " + vers +
" (expected A.B.* format)");
}
// Special handling for Hadoop 1.x and 2.x
switch (Integer.parseInt(parts[0])) {
case 0:
break;
case 1:
return "0.20S";
case 2:
return "0.23";
default:
throw new IllegalArgumentException("Unrecognized Hadoop major version number: " + vers);
}
String majorVersion = parts[0] + "." + parts[1];
// If we are running a security release, we won't have UnixUserGroupInformation
// (removed by HADOOP-6299 when switching to JAAS for Login)
try {
Class.forName("org.apache.hadoop.security.UnixUserGroupInformation" );
} catch (ClassNotFoundException cnf) {
if ("0.20".equals(majorVersion)) {
majorVersion += "S";
}
}
return majorVersion;
}
而不同的shims中规定了不同版本下的属性和方法等,比如判断是否localmode:
1). Hadoop20SShims 类(Hadoop 0.20 with Security和1.x版本 )
@Override
public boolean isLocalMode(Configuration conf) {
return "local".equals(getJobLauncherRpcAddress(conf));
}
@Override
public String getJobLauncherRpcAddress( Configuration conf) { //通过判断mapred.job.tracker的设置
return conf.get( "mapred.job.tracker");
}
2).Hadoop20Shims类 (Hadoop 0.20版本) 同上
3).Hadoop23Shims (Hadoop 2.x版本)
@Override
public boolean isLocalMode( Configuration conf) {
return "local".equals(conf.get("mapreduce.framework.name" )); //根据mapreduce.framework.name设置
}
转载于:https://blog.51cto.com/caiguangguang/1564601