sqoop mysql导入hive报错_[Hadoop] Sqoop 从MySQL导入数据到hive表报错“Could not load org.apache.hadoop.hive.conf.Hiv...

Sqoop从MySQL导入数据到Hive表报错：ERROR hive.HiveConfig: Could not load org.apache.hadoop.hive.conf.HiveConf. Make sure HIVE_CONF_DIR is set correctly.

[hadoop@hadoop01 ~]$ sqoop create-hive-table --connect jdbc:mysql://localhost:3306/db_1 --username root --password root --table city_info --hive-table city_info --fields-terminated-by ',' --hive-overwrite;

Warning: /home/hadoop/app/sqoop-1.4.6-cdh5.7.0/../hbase does not exist! HBase imports will fail.

Please set $HBASE_HOME to the root of your HBase installation.

Warning: /home/hadoop/app/sqoop-1.4.6-cdh5.7.0/../hcatalog does not exist! HCatalog jobs will fail.

Please set $HCAT_HOME to the root of your HCatalog installation.

Warning: /home/hadoop/app/sqoop-1.4.6-cdh5.7.0/../accumulo does not exist! Accumulo imports will fail.

Please set $ACCUMULO_HOME to the root of your Accumulo installation.

Warning: /home/hadoop/app/sqoop-1.4.6-cdh5.7.0/../zookeeper does not exist! Accumulo imports will fail.

Please set $ZOOKEEPER_HOME to the root of your Zookeeper installation.

18/06/26 22:39:46 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6-cdh5.7.0

18/06/26 22:39:46 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.

18/06/26 22:39:46 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.

18/06/26 22:39:47 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `city_info` AS t LIMIT 1

18/06/26 22:39:47 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `city_info` AS t LIMIT 1

18/06/26 22:39:48 INFO hive.HiveImport: Loading uploaded data into Hive

18/06/26 22:39:48 ERROR hive.HiveConfig: Could not load org.apache.hadoop.hive.conf.HiveConf. Make sure HIVE_CONF_DIR is set correctly.

18/06/26 22:39:48 ERROR tool.CreateHiveTableTool: Encountered IOException running create table job: java.io.IOException: java.lang.ClassNotFoundException: org.apache.hadoop.hive.conf.HiveConf

at org.apache.sqoop.hive.HiveConfig.getHiveConf(HiveConfig.java:50)

at org.apache.sqoop.hive.HiveImport.getHiveArgs(HiveImport.java:392)

at org.apache.sqoop.hive.HiveImport.executeExternalHiveScript(HiveImport.java:379)

at org.apache.sqoop.hive.HiveImport.executeScript(HiveImport.java:337)

at org.apache.sqoop.hive.HiveImport.importTable(HiveImport.java:241)

at org.apache.sqoop.tool.CreateHiveTableTool.run(CreateHiveTableTool.java:58)

at org.apache.sqoop.Sqoop.run(Sqoop.java:143)

at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)

at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179)

at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218)

at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227)

at org.apache.sqoop.Sqoop.main(Sqoop.java:236)

Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hive.conf.HiveConf

at java.net.URLClassLoader.findClass(URLClassLoader.java:381)

at java.lang.ClassLoader.loadClass(ClassLoader.java:424)

at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)

at java.lang.ClassLoader.loadClass(ClassLoader.java:357)

at java.lang.Class.forName0(Native Method)

at java.lang.Class.forName(Class.java:264)

at org.apache.sqoop.hive.HiveConfig.getHiveConf(HiveConfig.java:44)

... 11 more

[hadoop@hadoop01 ~]$

解决：

# 查看HiveConf.class类是否存在

[hadoop@hadoop01 app]$ cd apache-hive-1.1.0-cdh5.7.0-bin/lib/

[hadoop@hadoop01 lib]$ jar tf hive-common-1.1.0-cdh5.7.0.jar | grep HiveConf.class

org/apache/hadoop/hive/conf/HiveConf.class

[hadoop@hadoop01 lib]$

查看到HiveConf.class类明明存在，只是环境没有找到。

修改环境配置，将hive的lib添加HADOOP_CLASSPATH中

[root@hadoop01 ~]# vi /etc/profile

export HADOOP_CLASSPATH=/home/hadoop/app/hadoop-2.6.0-cdh5.7.0/lib/*

export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:/home/hadoop/app/apache-hive-1.1.0-cdh5.7.0-bin/lib/*

再次执行，执行成功。

[hadoop@hadoop01 ~]$ sqoop create-hive-table --connect jdbc:mysql://localhost:3306/db_1 --username root --password root --table city_info --hive-table city_info --fields-terminated-by ',' --hive-overwrite;

Warning: /home/hadoop/app/sqoop-1.4.6-cdh5.7.0/../hbase does not exist! HBase imports will fail.

Please set $HBASE_HOME to the root of your HBase installation.

Warning: /home/hadoop/app/sqoop-1.4.6-cdh5.7.0/../hcatalog does not exist! HCatalog jobs will fail.

Please set $HCAT_HOME to the root of your HCatalog installation.

Warning: /home/hadoop/app/sqoop-1.4.6-cdh5.7.0/../accumulo does not exist! Accumulo imports will fail.

Please set $ACCUMULO_HOME to the root of your Accumulo installation.

Warning: /home/hadoop/app/sqoop-1.4.6-cdh5.7.0/../zookeeper does not exist! Accumulo imports will fail.

Please set $ZOOKEEPER_HOME to the root of your Zookeeper installation.

18/06/26 23:19:58 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6-cdh5.7.0

18/06/26 23:19:58 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.

18/06/26 23:19:58 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.

18/06/26 23:19:59 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `city_info` AS t LIMIT 1

18/06/26 23:19:59 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `city_info` AS t LIMIT 1

18/06/26 23:20:00 INFO hive.HiveImport: Loading uploaded data into Hive

Logging initialized using configuration in jar:file:/home/hadoop/app/apache-hive-1.1.0-cdh5.7.0-bin/lib/hive-common-1.1.0-cdh5.7.0.jar!/hive-log4j.properties

OK

Time taken: 1.427 seconds

[hadoop@hadoop01 ~]$