sqoop 抽取源码流程分析(一) 主流程分析以及各种插件

1. 工具类的继承关系

目前sqoop 提供的各种工具，比如 ImportTool, ExportTool, CodeGenTool 等工具类都是集成于 BaseSqoopTool 这个基础类
下面看看这个BaseSqoopTool 基础类的继承关系

public abstract class BaseSqoopTool 
   extends com.cloudera.sqoop.tool.SqoopTool {
......
public abstract class SqoopTool

    extends org.apache.sqoop.tool.SqoopTool {

......

public abstract class SqoopTool {

2. sqoop 之 main 函数

2.1 sqoop 继承关系

public class Sqoop extends Configured implements Tool {
......

其中的Tool 和 Configured 是hadoop 中的基础类，毕竟sqoop 之用hadoop 来做的具体实现。

2.2 主函数

 public static void main(String [] args) {

    if (args.length == 0) {

      System.err.println("Try 'sqoop help' for usage.");

      System.exit(1);

    }

    int ret = runTool(args);

    System.exit(ret);

  }

2.3 runTool 的实现

runTool 更具传入的sqoop 参数，选择需要使用的工具类名称来构建相应的工具类

......
   SqoopTool tool = SqoopTool.getTool(toolName);
......
   Sqoop sqoop = new Sqoop(tool, pluginConf);

    return runSqoop(sqoop, Arrays.copyOfRange(expandedArgs, 1, expandedArgs.length));

那么具体的运行就是在各个工具类的具体实现里面了。
具体的实现可以参考对importTool的具体分析了

2.4 隐藏的关系

 public static int runSqoop(Sqoop sqoop, String [] args) {

    try {

      String [] toolArgs = sqoop.stashChildPrgmArgs(args);

      return ToolRunner.run(sqoop.getConf(), sqoop, toolArgs);

......

虽然是在hadoop 里面实现的这个run 方法，不过可以通过下面的具体实现来看出，其实调用的还是sqoop的run方法

 public static int run(Configuration conf, Tool tool, String[] args) 

    throws Exception{

    if(conf == null) {

      conf = new Configuration();

    }

    GenericOptionsParser parser = new GenericOptionsParser(conf, args);

    //set the configuration back, so that Tool can configure itself

    tool.setConf(conf);



    //get the args w/o generic hadoop args

    String[] toolArgs = parser.getRemainingArgs();

    return tool.run(toolArgs);

2.5 真正的入口点

 public int run(String [] args) {

    if (options.getConf() == null) {

      // Configuration wasn't initialized until after the ToolRunner

      // got us to this point. ToolRunner gave Sqoop itself a Conf

      // though.

      options.setConf(getConf());

    }



    try {

      options = tool.parseArguments(args, null, options, false);

      tool.appendArgs(this.childPrgmArgs);

      tool.validateOptions(options);

    } catch (Exception e) {

      // Couldn't parse arguments.

      // Log the stack trace for this exception

      LOG.debug(e.getMessage(), e);

      // Print exception message.

      System.err.println(e.getMessage());

      return 1; // Exit on exception here.

    }



    return tool.run(options);

  }

2.5.1 参数解析

public SqoopOptions parseArguments(String [] args,

      Configuration conf, SqoopOptions in, boolean useGenericOptions)

      throws ParseException, SqoopOptions.InvalidOptionsException {

......

 // Parse tool-specific arguments.

    ToolOptions toolOptions = new ToolOptions();

    configureOptions(toolOptions);

    CommandLineParser parser = new SqoopParser();

    CommandLine cmdLine = parser.parse(toolOptions.merge(), toolArgs, true);

    applyOptions(cmdLine, out);

    this.extraArguments = cmdLine.getArgs();

可以看出真正的参数解析是在SqoopParser这个类中

那么继续看SqoopParser 的parse方法

 public void processArgs(Option opt, ListIterator iter)

      throws ParseException {

    // Loop until an option is found.

    while (iter.hasNext()) {

      String str = (String) iter.next();

      if (getOptions().hasOption(str) && str.startsWith("-")) {

        // found an Option, not an argument.

        iter.previous();

        break;

      }



      // Otherwise, this is a value.

      try {

        // Note that we only strip matched quotes here.

        addValForProcessing.invoke(opt, stripMatchedQuotes(str));

      } catch (IllegalAccessException iae) {

        throw new RuntimeException(iae);

      } catch (java.lang.reflect.InvocationTargetException ite) {

        // Any runtime exception thrown within addValForProcessing()

        // will be wrapped in an InvocationTargetException.

        iter.previous();

        break;

      } catch (RuntimeException re) {

        iter.previous();

        break;

      }

    }



    if (opt.getValues() == null && !opt.hasOptionalArg()) {

      throw new MissingArgumentException(opt);

    }

  }