2021SC@SDUSC
本次我将对hadoop中的common部分中Configuration最后一部分内容进行解析:
addResourceObject以及若干方法
实际上,在hadoop中无论使用何种addResource,最终都是调用了addResourceObject(Resource resource)的方法,这个方法会先将资源添加到一个全局的List集合中,然后调用reloadConfiguration来触发刷新properties进而使标记为final的key失效。
findSubVariable subsituteVars
在hadoop-2.7版本之前,只有一个substituteVars方法,使用java自身的正则表达式来匹配获得${user.home }
中间的值(user.home)。
hadoop-2.7版本之后,为了提升性能,hadoop实现了匹配获取中间值的方法( findSubVariable
) ps:可能是由于java自身的正则表达式方式消耗性能过大,所以,通过自己内部方法手动匹配,降低性能的消耗。
该方法将字符串中¥{}中间位置的区间获取到
private static int[] findSubVariable(String eval) {
int[] result = {-1, -1};
int matchStart;
int leftBrace;
// scanning for a brace first because it's less frequent than $
// that can occur in nested class names
//
match_loop:
for (matchStart = 1, leftBrace = eval.indexOf('{', matchStart);
// minimum left brace position (follows '$')
leftBrace > 0
// right brace of a smallest valid expression "${c}"
&& leftBrace + "{c".length() < eval.length();
leftBrace = eval.indexOf('{', matchStart)) {
int matchedLen = 0;
if (eval.charAt(leftBrace - 1) == '$') {
int subStart = leftBrace + 1; // after '{'
for (int i = subStart; i < eval.length(); i++) {
switch (eval.charAt(i)) {
case '}':
if (matchedLen > 0) { // match
result[SUB_START_IDX] = subStart;
result[SUB_END_IDX] = subStart + matchedLen;
break match_loop;
}
// fall through to skip 1 char
case ' ':
case '$':
matchStart = i + 1;
continue match_loop;
default:
matchedLen++;
}
}
// scanned from "${" to the end of eval, and no reset via ' ', '$':
// no match!
break match_loop;
} else {
// not a start of a variable
//
matchStart = leftBrace + 1;
}
}
return result;
}
获取key进行替换,若System.getProperty()存在,则进行替换。若不存在,则查找properties,如果存在进行替换,不存在则原样保留。
private String substituteVars(String expr) {
if (expr == null) {
return null;
}
String eval = expr;
for(int s = 0; s < MAX_SUBST; s++) {
final int[] varBounds = findSubVariable(eval);
if (varBounds[SUB_START_IDX] == -1) {
return eval;
}
final String var = eval.substring(varBounds[SUB_START_IDX],
varBounds[SUB_END_IDX]);
String val = null;
if (!restrictSystemProps) {
try {
if (var.startsWith("env.") && 4 < var.length()) {
String v = var.substring(4);
int i = 0;
for (; i < v.length(); i++) {
char c = v.charAt(i);
if (c == ':' && i < v.length() - 1 && v.charAt(i + 1) == '-') {
val = getenv(v.substring(0, i));
if (val == null || val.length() == 0) {
val = v.substring(i + 2);
}
break;
} else if (c == '-') {
val = getenv(v.substring(0, i));
if (val == null) {
val = v.substring(i + 1);
}
break;
}
}
if (i == v.length()) {
val = getenv(v);
}
} else {
val = getProperty(var);
}
} catch (SecurityException se) {
LOG.warn("Unexpected SecurityException in Configuration", se);
}
}
if (val == null) {
val = getRaw(var);
}
if (val == null) {
return eval; // return literal ${var}: var is unbound
}
final int dollar = varBounds[SUB_START_IDX] - "${".length();
final int afterRightBrace = varBounds[SUB_END_IDX] + "}".length();
final String refVar = eval.substring(dollar, afterRightBrace);
// detect self-referential values
if (val.contains(refVar)) {
return expr; // return original expression if there is a loop
}
// substitute
eval = eval.substring(0, dollar)
+ val
+ eval.substring(afterRightBrace);
}
throw new IllegalStateException("Variable substitution depth too large: "
+ MAX_SUBST + " " + expr);
}
set方法
通过程序对key-value进行设置,source允许为空,若用户不设置源,则程序自动将programatically
当值为被遗弃的,则该方法会先将新的key得到,并设置source为 because old key is deprecated
public void set(String name, String value, String source) {
Preconditions.checkArgument(
name != null,
"Property name must not be null");
Preconditions.checkArgument(
value != null,
"The value of property %s must not be null", name);
name = name.trim();
DeprecationContext deprecations = deprecationContext.get();
if (deprecations.getDeprecatedKeyMap().isEmpty()) {
getProps();
}
getOverlay().setProperty(name, value);
getProps().setProperty(name, value);
String newSource = (source == null ? "programmatically" : source);
if (!isDeprecated(name)) {
putIntoUpdatingResource(name, new String[] {newSource});
String[] altNames = getAlternativeNames(name);
if(altNames != null) {
for(String n: altNames) {
if(!n.equals(name)) {
getOverlay().setProperty(n, value);
getProps().setProperty(n, value);
putIntoUpdatingResource(n, new String[] {newSource});
}
}
}
}
else {
String[] names = handleDeprecation(deprecationContext.get(), name);
String altSource = "because " + name + " is deprecated";
for(String n : names) {
getOverlay().setProperty(n, value);
getProps().setProperty(n, value);
putIntoUpdatingResource(n, new String[] {altSource});
}
}
}
loadResource
该方法用来解析xml,同时采用了DOM解析,支持xml文件的引入
和之前的版本相比较,xml配置文件中,在property中可以声明source标签,声明资源的信息。