Sqoop从MySQL导入数据到Hive表报错:ERROR hive.HiveConfig: Could not load org.apache.hadoop.hive.conf.HiveConf. Make sure HIVE_CONF_DIR is set correctly.
[hadoop@hadoop01 ~]$ sqoop create-hive-table --connect jdbc:mysql://localhost:3306/db_1 --username root --password root --table city_info --hive-table city_info --fields-terminated-by ',' --hive-overwrite;
Warning: /home/hadoop/app/sqoop-1.4.6-cdh5.7.0/../hbase does not exist! HBase imports will fail.
Please set $HBASE_HOME to the root of your HBase installation.
Warning: /home/hadoop/app/sqoop-1.4.6-cdh5.7.0/../hcatalog does not exist! HCatalog jobs will fail.
Please set $HCAT_HOME to the root of your HCatalog installation.
Warning: /home/hadoop/app/sqoop-1.4.6-cdh5.7.0/../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
Warning: /home/hadoop/app/sqoop-1.4.6-cdh5.7.0/../zookeeper does not exist! Accumulo imports will fail.
Please set $ZOOKEEPER_HOME to the root of your Zookeeper installation.
18/06/26 22:39:46 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6-cdh5.7.0
18/06/26 22:39:46 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
18/06/26 22:39:46 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
18/06/26 22:39:47 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `city_info` AS t LIMIT 1
18/06/26 22:39:47 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `city_info` AS t LIMIT 1
18/06/26 22:39:48 INFO hive.HiveImport: Loading uploaded data into Hive
18/06/26 22:39:48 ERROR hive.HiveConfig: Could not load org.apache.hadoop.hive.conf.HiveConf. Make sure HIVE_CONF_DIR is set correctly.
18/06/26 22:39:48 ERROR tool.CreateHiveTableTool: Encountered IOException running create table job: java.io.IOException: java.lang.ClassNotFoundException: org.apache.hadoop.hive.conf.HiveConf
at org.apache.sqoop.hive.HiveConfig.getHiveConf(HiveConfig.java:50)
at org.apache.sqoop.hive.HiveImport.getHiveArgs(HiveImport.java:392)
at org.apache.sqoop.hive.HiveImport.executeExternalHiveScript(HiveImport.java:379)
at org.apache.sqoop.hive.HiveImport.executeScript(HiveImport.java:337)
at org.apache.sqoop.hive.HiveImport.importTable(HiveImport.java:241)
at org.apache.sqoop.tool.CreateHiveTableTool.run(CreateHiveTableTool.java:58)
at org.apache.sqoop.Sqoop.run(Sqoop.java:143)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227)
at org.apache.sqoop.Sqoop.main(Sqoop.java:236)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hive.conf.HiveConf
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:264)
at org.apache.sqoop.hive.HiveConfig.getHiveConf(HiveConfig.java:44)
... 11 more
[hadoop@hadoop01 ~]$
解决:
# 查看HiveConf.class类是否存在
[hadoop@hadoop01 app]$ cd apache-hive-1.1.0-cdh5.7.0-bin/lib/
[hadoop@hadoop01 lib]$ jar tf hive-common-1.1.0-cdh5.7.0.jar | grep HiveConf.class
org/apache/hadoop/hive/conf/HiveConf.class
[hadoop@hadoop01 lib]$
查看到HiveConf.class类明明存在,只是环境没有找到。
修改环境配置,将hive的lib添加HADOOP_CLASSPATH中
[root@hadoop01 ~]# vi /etc/profile
export HADOOP_CLASSPATH=/home/hadoop/app/hadoop-2.6.0-cdh5.7.0/lib/*
export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:/home/hadoop/app/apache-hive-1.1.0-cdh5.7.0-bin/lib/*
再次执行,执行成功。
[hadoop@hadoop01 ~]$ sqoop create-hive-table --connect jdbc:mysql://localhost:3306/db_1 --username root --password root --table city_info --hive-table city_info --fields-terminated-by ',' --hive-overwrite;
Warning: /home/hadoop/app/sqoop-1.4.6-cdh5.7.0/../hbase does not exist! HBase imports will fail.
Please set $HBASE_HOME to the root of your HBase installation.
Warning: /home/hadoop/app/sqoop-1.4.6-cdh5.7.0/../hcatalog does not exist! HCatalog jobs will fail.
Please set $HCAT_HOME to the root of your HCatalog installation.
Warning: /home/hadoop/app/sqoop-1.4.6-cdh5.7.0/../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
Warning: /home/hadoop/app/sqoop-1.4.6-cdh5.7.0/../zookeeper does not exist! Accumulo imports will fail.
Please set $ZOOKEEPER_HOME to the root of your Zookeeper installation.
18/06/26 23:19:58 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6-cdh5.7.0
18/06/26 23:19:58 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
18/06/26 23:19:58 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
18/06/26 23:19:59 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `city_info` AS t LIMIT 1
18/06/26 23:19:59 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `city_info` AS t LIMIT 1
18/06/26 23:20:00 INFO hive.HiveImport: Loading uploaded data into Hive
Logging initialized using configuration in jar:file:/home/hadoop/app/apache-hive-1.1.0-cdh5.7.0-bin/lib/hive-common-1.1.0-cdh5.7.0.jar!/hive-log4j.properties
OK
Time taken: 1.427 seconds
[hadoop@hadoop01 ~]$