Sqoop学习笔记

Sqoop学习笔记

1）列出数据库
sqoop list-databases –connect jdbc:mysql://localhost/ -username root -P

2）将数据库导入HDFS ：
sqoop import –connect jdbc:mysql://localhost/mytest_sqoop –table widgets -m 2 -username root -P
-m 2:使用两个map任务
输出结果存在当前命令行目录中，包括输出文件和生成代码widget.java

3）将Mysql数据一步导入Hive
sqoop import –connect jdbc:mysql://localhost/mytest_sqoop –table widgets -m 2 –hive-import -username root -P
将创建widgets的hive表和数据集

4）导出：从HDFS 导出到远程的数据库目标
必须首先在数据库中创建用于接收的目标表

错误解决方法

导入MySql数据表时候遇到如下错误：
sqoop import –connect jdbc:mysql://localhost/mytest_sqoop –table widgets -m -1
（1）Got exception running Sqoop: java.lang.RuntimeException: Could not load db driver class: com.mysql.jdbc.Driver
无法加载数据库驱动:解决方法，将驱动jar放到sqoop/lib目录下：如
cp mysql-connector-java-5.1.6-bin.jar ~/hadoop/sqoop/lib/mysql-connector-java-5.1.6-bin.jar

(2)权限问题 ，加上：
–username root –P

(3)mysql驱动版本问题：
ERROR manager.SqlManager: Error reading from database: java.sql.SQLException: Streaming result set com.mysql.jdbc.RowDataDynamic@6c4fc156 is still active. No statements may be issued when any streaming result sets are open and in use on a given connection. Ensure that you have called .close() on any active streaming result sets before attempting more queries.
java.sql.SQLException: Streaming result set com.mysql.jdbc.RowDataDynamic@6c4fc156 is still active. No statements may be issued when any streaming result sets are open and in use on a given connection. Ensure that you have called .close() on any active streaming result sets before attempting more queries.
mysql-connect-java jar包的版本不对
改为mysql-connector-java-5.1.31 这个版本后就可以了