-
在开始学习hadoop时,最痛苦的一件事就是难以理解所写程序的执行过程,让我们先来看这个实例,这个测试类ToolRunnerTest继承Configured的基础上实现了Tool接口,下面对其用到的基类源码进行分析,就可以理解其执行过程是如此简单。。。。。。
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.conf.Configured;
import org.apache.hadoop.util.Tool;
import org.apache.hadoop.util.ToolRunner;
public class ToolRunnerTest extends Configured implements Tool {
public int run(String[] arg0) throws Exception {
// 调用基类Configured的getConf获取环境变量实例
Configuration conf = getConf();
// 获取属性值
System.out.println("flower is " + conf.get("flower"));
System.out.println("color id " + conf.get("color"));
System.out.println("blossom ? " + conf.get("blossom"));
System.out.println("this is the host default name ="
+ conf.get("fs.default.name"));
return 0;
}
/**
* @param args
* @throws Exception
*/
public static void main(String[] args) throws Exception {
// 获取当前环境变量
Configuration conf = new Configuration();
// 使用ToolRunner的run方法对自定义的类型进行处理
ToolRunner.run(conf, new ToolRunnerTest(), args);
}
}基类Configured实现了Configurable接口,而Configurable接口源码如下
{
public abstract void setConf(Configuration configuration);
public abstract Configuration getConf();
}Configured则必须实现Configurable类的两个方法,源码如下
implements Configurable
{
public Configured()
{
this(null);
}
public Configured(Configuration conf)
{
setConf(conf);
}
public void setConf(Configuration conf)
{
this.conf = conf;
}
public Configuration getConf()
{
return conf;
}
private Configuration conf;
}Tool的源码如下所示:
public interface Tool extends Configurable {
int run(String [] args) throws Exception;
}
就这么一点点
ToolRunner类的源码如下
{
public ToolRunner()
{
}
public static int run(Configuration conf, Tool tool, String args[])
throws Exception
{
if(conf == null)
conf = new Configuration();
GenericOptionsParser parser = new GenericOptionsParser(conf, args);
tool.setConf(conf);
String toolArgs[] = parser.getRemainingArgs();
return tool.run(toolArgs);
}
public static int run(Tool tool, String args[])
throws Exception
{
return run(tool.getConf(), tool, args);
}
public static void printGenericCommandUsage(PrintStream out)
{
GenericOptionsParser.printGenericCommandUsage(out);
}
public static boolean confirmPrompt(String prompt)
throws IOException
{
do
{
...
} while(true);
}
}解析:当程序执行ToolRunner.run(conf, new ToolRunnerTest(), args);时,会转到ToolRunner类的run方法部分,因为Configuration已经实例,所以直至执行到tool.run(toolArgs);又因为Tool是一个只含有一个run方法框架的接口,所以将执行实现这个接口的类ToolRunnerTest的run方法。完成其输出。其实在看完这几个类的源码后,其执行过程是很简单的
该实例的运行结果如下:
master:/opt>hadoop jar ToolRunnerTest.jar
flower is null
color id null
blossom ? null
14/09/30 10:13:21 INFO Configuration.deprecation: fs.default.name is deprecated. Instead, use fs.defaultFS
this is the host default name =hdfs://master:8020
-
源地址:http://www.it165.net/admin/html/201410/3821.html
-
Hadoop中的辅助类ToolRunner和Configured的用法详解
最新推荐文章于 2021-03-21 15:59:54 发布