代码连接sqoop 导入hdfs 遇到的一些问题

最近有要用到sqoop，但是不能以脚本的形式。研究了一下，中途有遇到一些坑，所幸最后成功，
在这里记录一下，自己做个笔记，也希望能帮助到别人。
首先说下我的集群环境以及mysql版本
集群： hadoop 版本3.0
MySQL 5.7
sqoop 版本 maven坐标中显示的版本是1.4.6
这里需要注意的是 maven中的jar是下载不下来的，我个人自作聪明的从maven 仓库手动下载，然后 自个 建立本地坐标（事实上这个是一个坑，我们稍后说明）
maven 坐标为

org.apache.sqoop
sqoop
1.4.6

刚才说maven依赖是下载不下来的，我们拿到 sqoop的安装包 （sqoop-1.4.6.bin__hadoop-2.0.4-alpha.tar.gz）解压 将进入刚才解压的目录，复制sqoop-1.4.6.jar 到maven仓库响应的坐标下。

下边上代码

import org.apache.hadoop.conf.Configuration;
import org.apache.sqoop.Sqoop;
import org.apache.sqoop.tool.SqoopTool;
import org.apache.sqoop.util.OptionsFileUtil;

/**
 * @author roman
 * @version 1.0
 * @description
 * @date 2021/3/26 14:17
 */

public class Demo {


        public int importDataFromOracle() {
        String[] args = new String[]{
                "--connect", "jdbc:oracle:thin:@192.168.10.12:1521:ftmsdev",
                "--username", "ftms",
                "--password", "ftms",
                "--table", "SYSTEM_USER",
                "--columns", "LOGIN_NAME,PASSWORD",
                "--split-by", "USER_ID",
                "--target-dir", "ftms/tables/system_user"
        };

        String[] expandedArgs = null;
        try {
            expandedArgs = OptionsFileUtil.expandArguments(args);
        } catch (Exception e){
            System.err.println(e.getMessage());
            System.err.println("Try 'sqoop help' for usage.");
        }

        com.cloudera.sqoop.tool.SqoopTool tool = (com.cloudera.sqoop.tool.SqoopTool) SqoopTool.getTool("import");
        //com.cloudera.sqoop.tool.SqoopTool tool = new ImportTool();

        Configuration conf = new Configuration();
        conf.set("fs.default.name", "hdfs://192.168.242.128:9000");//设置hadoop服务地址
        Configuration pluginConf =null;// SqoolTool.loadPlugins(conf);

        Sqoop sqoop = new Sqoop(tool, pluginConf);
        return Sqoop.runSqoop(sqoop, expandedArgs);
    }
    private static int importData() throws Exception {
        String[] args = new String[]{
                "--connect", "jdbc:mysql://localhost:3306/datae?characterEncoding=utf-8&serverTimezone=UTC",
                "--driver", "com.mysql.jdbc.Driver",
                "--username", "root",
                "--password", "123456",
                "--table", "t_dept",
                "--target-dir", "/detp1",
                "--delete-target-dir",
                "--num-mappers","1"
        };

        String[] expandArguments = OptionsFileUtil.expandArguments(args);

        SqoopTool tool = SqoopTool.getTool("import");

        Configuration conf = new Configuration();
        // 设置hdfs 的namenode的通信地址 ，需要注意的是原生的hdfs端口是9000，此时我的是cdh版本的
        conf.set("fs.default.name", "hdfs://192.168.35.11:8020");
        Configuration loadPlugins = SqoopTool.loadPlugins(conf);

        Sqoop sqoop = new Sqoop((com.cloudera.sqoop.tool.SqoopTool) tool, loadPlugins);
        return Sqoop.runSqoop(sqoop, expandArguments);
    }
}

经过以上操作，应该能成功将数据写入到hdfs中。

遇到的问题

此时会出现一些奇怪的现象，比如我本例中数据库是datae,sqoop会报dataexplorer找不到（但是数据，也就是datae我配置的表中数据切切实实的导入到了hdfs --（hdfs 查看命令 hadoop fs --ls /dept1）），此问题我尚未解决，希望有缘人能解决。

atLast

刚才说为什么不能用maven仓库的jar 因为用了之后 ，就会报 如下异常

Exception in thread "main" java.lang.IncompatibleClassChangeError: Found interface org.apache.hadoop.mapreduce.JobContext, but class was expected
	at org.apache.sqoop.config.ConfigurationHelper.getJobNumMaps(ConfigurationHelper.java:65)
	at com.cloudera.sqoop.config.ConfigurationHelper.getJobNumMaps(ConfigurationHelper.java:36)
	at org.apache.sqoop.mapreduce.db.DataDrivenDBInputFormat.getSplits(DataDrivenDBInputFormat.java:125)
	at org.apache.hadoop.mapreduce.JobSubmitter.writeNewSplits(JobSubmitter.java:314)
	at org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:331)
	at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:202)
	at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1570)
	at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1567)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1926)
	at org.apache.hadoop.mapreduce.Job.submit(Job.java:1567)
	at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1588)
	at org.apache.sqoop.mapreduce.ImportJobBase.doSubmitJob(ImportJobBase.java:196)
	at org.apache.sqoop.mapreduce.ImportJobBase.runJob(ImportJobBase.java:169)
	at org.apache.sqoop.mapreduce.ImportJobBase.runImport(ImportJobBase.java:266)
	at org.apache.sqoop.manager.SqlManager.importTable(SqlManager.java:673)
	at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:497)
	at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:605)
	at org.apache.sqoop.Sqoop.run(Sqoop.java:143)
	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
	at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179)
	at Demo.importData(Demo.java:67)
	at Demo.main(Demo.java:71)