mysql和sql互导_用Sqoop进行MySQL和HDFS系统间的数据互导

sqoop 是apache下用于RDBMS和HDFS互相导数据的工具。

本文档是sqoop的使用实例，实现从MySQL到hdfs互导数据，以及从Mysql导数据到HBase。

下载：http://www.apache.org/dyn/closer.cgi/sqoop/

[zhouhh@Hadoop48 ~]$ wget http://labs.renren.com/apache-mirror/sqoop/1.4.1-incubating/sqoop-1.4.1-incubating__hadoop-1.0.0.tar.gz

最新用户手册

http://sqoop.apache.org/docs/1.4.1-incubating/SqoopUserGuide.html

一、从HBase库中直接导出到mysql中？

一开始我想从HBase库中直接导出到mysql中。

在mysql中创建一个库和表

mysql> create database toplists;

Query OK, 1 row affected (0.06 sec)

mysql> use toplists

Database changed

mysql> create table t1(id int not null primary key, name varchar(255),value int);

Query OK, 0 rows affected (0.10 sec)

hbase(main):011:0> scan 't1'

ROW COLUMN+CELL

1001 column=info:count, timestamp=1340265059531, value=724988

1009 column=info:count, timestamp=1340265059533, value=108051

...

total column=info:count, timestamp=1340265059534, value=833039

total_user_count column=info:, timestamp=1340266656307, value=154516

11 row(s) in 0.0420 seconds

[zhouhh@Hadoop48 ~]$ sqoop list-tables --connect jdbc:mysql://localhost/toplists --username root

Java.lang.RuntimeException: Could not load db driver class: com.mysql.jdbc.Driver

at org.apache.sqoop.manager.SqlManager.makeConnection(SqlManager.java:657)

at org.apache.sqoop.manager.GenericJdbcManager.getConnection(GenericJdbcManager.java:52)

at org.apache.sqoop.manager.SqlManager.execute(SqlManager.java:473)

at org.apache.sqoop.manager.SqlManager.execute(SqlManager.java:496)

at org.apache.sqoop.manager.SqlManager.getColumnTypesForRawQuery(SqlManager.java:194)

at org.apache.sqoop.manager.SqlManager.getColumnTypes(SqlManager.java:178)

at org.apache.sqoop.manager.ConnManager.getColumnTypes(ConnManager.java:114)

at org.apache.sqoop.orm.ClassWriter.getColumnTypes(ClassWriter.java:1235)

at org.apache.sqoop.orm.ClassWriter.generate(ClassWriter.java:1060)

at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:82)

at org.apache.sqoop.tool.ExportTool.exportTable(ExportTool.java:64)

at org.apache.sqoop.tool.ExportTool.run(ExportTool.java:97)

at org.apache.sqoop.Sqoop.run(Sqoop.java:145)

at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)

at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:181)

at org.apache.sqoop.Sqoop.runTool(Sqoop.java:220)

at org.apache.sqoop.Sqoop.runTool(Sqoop.java:229)

at org.apache.sqoop.Sqoop.main(Sqoop.java:238)

at com.cloudera.sqoop.Sqoop.main(Sqoop.java:57)