sqoop查询oracle数据报表不存在,使用sqoop1.4.4从oracle导入数据到hive中错误记录及解决方案...

在使用命令导数据过程中，出现如下错误

sqoop import --hive-import --connect jdbc:oracle:thin:@192.168.29.16:1521/testdb --username NAME --passord PASS --verbose -m 1 --table T_USERINFO

错误1：File does not exist: hdfs://opt/sqoop-1.4.4/lib/commons-io-1.4.jar

FileNotFoundException: File does not exist: hdfs://opt/sqoop-1.4.4/lib/commons-io-1.4.jar at org.apache ... ...

at org.apache ... ...

原因分析:

It is common for Hadoop services to look for jars in HDFS because all nodes in the cluster can access files in HDFS. This is important if the MapReduce job being kicked off by the Hadoop service, in this caseSqoop, has a dependence on those jars. Remember, the Mappers are running on a DataNode, not the NameNode even though you are (probably) running the Sqoop command from the NameNode. Putting the jars on HDFS is not the only possible solution to this problem, but it is a sensible one.

Now we can deal with the actual error. At least one, but probably all, of your Mappers are unable tofind a jar they need. That means that either the jar does not exist or the user trying to access them does not have the required permissions. First check if the file exists by running hadoop fs -ls home/SqoopUser/sqoop-1.4.3-cdh4.4.0/sqoop-1.4.3-cdh4.4.0.jar by a user with superuser privileges on the cluster. If it does not exist, put it there with hadoop fs -put {jarLocationOn/NameNode/fileSystem/sqoop-1.4.3-cdh4.4.0.jar} /home/SqoopUser/sqoop-1.4.3-cdh4.4.0/sqoop-1.4.3-cdh4.4.0.jar.

解决方法：

将提示中涉及的jar文件put到hdfs文件系统中的相同位置，如果文件系统中没有对应的目录，则需要建立相应目录，在我的错误提示中，由于hdfs://master:8020/中缺少了 /opt/sqoop-1.4.4/lib/文件夹中的各种jar,所以我的做法是把此处整个/opt/sqoop-1.4.4/lib文件夹put到hdfs://master:8020/中

hadoop fs-ls -R /

hadoop fs-mkdir /opt

hadoop fs-mkdir /opt/sqoop-1.4.4

hadoop fs-put /opt/sqoop-1.4.4/lib /opt/sqoop-1.4.4/

hadoop fs-ls -R /opt/sqoop-1.4.4

错误2 ：java.lang.ClassNotFoundException: Class U_BASICINFO not found

对于要导入到hive中的表，错误提示说找不到对应的.class和.jar文件

java.lang.Exception: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class U_BASICINFO not found

at org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:462)

at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:522)

Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class U_BASICINFO not found

at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895)

at org.apache.sqoop.mapreduce.db.DBConfiguration.getInputClass(DBConfiguration.java:394)

at .....

原因分析：暂时不知道

解决方案：

默认sqoop在执行导入table过程中会生成对应的table的java文件和编译产生的.class和.jar文件，.java文件保存在sqoop/bin目录下，而class 和 jar文件则保存在/tmp/sqoop-hduser/compile/ 下相应的文件夹中。

我的解决方式是找到要导入表的class和jar文件，然后将他们拷贝到sqoop/bin目录下面和hdfs文件系统中的/user/USERNAM/ 目录下面(后期测试后，只要将.class和.jar拷贝到sqoop/bin目录下就可以成功import)。

cp /tmp/sqoop-root/compile/某个临时文件夹包含需要的class和jar文件/*/opt/sqoop-1.4.4/bin/

hadoop fs -put /tmp/sqoop-root/compile/某个临时文件夹包含需要的class和jar文件/* /user/root/

错误3 org.apache.hadoop.mapred.file already exists exception:output directory hdfs://user/root/... ...

解决方案：

在执行过一次导入数据表命令后，当再次执行时，可能会出现这种错误，这是只要进入到hdfs中将对应的文件或者文件夹删除即可。

hadoop fs -rm /user/USERNAME/*

原文：http://www.cnblogs.com/wrencai/p/3908533.html