Load a mysql table to HDFS using Sqoop

最新推荐文章于 2024-01-20 22:18:50 发布

fdtsaid

最新推荐文章于 2024-01-20 22:18:50 发布

阅读量2k

点赞数

分类专栏： Hadoop

本文链接：https://blog.csdn.net/mozart_cai/article/details/8831249

版权

Hadoop 专栏收录该内容

6 篇文章 0 订阅

订阅专栏

[Target]

Load a mysql table to HDFS

[Environment]

Eucalyptus in cloud test;
1 cluster consist of node1, node2, node3 (all redhat6.2);
1 other machine node4 runs mysql services (centos6.2);
HDFS service running, Map-Reduce service running;

[Setup Mysql Service]

1.in node4

#Stop firewall

$ /etc/init.d/iptables stop
$ setenforce 0

#Install mysql and mysql server,

#if yum not work,
#set proxy "export http_proxy=proxy01.cd.intel.com:911"
$ yum list mysql
$ yum install mysql
$ yum list mysql-server
$ yum install mysql-server

#Start mysql service

#Create a Admin and try to login
$ /etc/init.d/mysqld start # or service mysqld start
$ mysqladmin -u root password 123456
$ mysql -u root -p
> hidden password

#Grant that we can use username:root, password:123456 to access mysql on node4 from any machine

$ sudo mysql -u root -p
#grant all on 数据库名.* to 用户名@'客户端IP地址' identified by '密码';
$ grant all on *.* to root@'%' identified by '123456';

2.in node3/node2/node1

#Try to access mysql service on node4
$ mysql -u root -h IP_Address_Of_Node4 -p

3.in node1/node2/node3/node4

#create demo table

$ create database hadoopguide;
$ create table mytable(id INT NOT NULL PRIMARY KEY AUTO_INCREMENT, design_comment VARCHAR(100));
$ INSERT INTO mytable VALUES (NULL, 'Connects two gizmos');
$ INSERT INTO mytable VALUES (NULL, NULL);
$ INSERT INTO mytable VALUES (NULL, 'Our flagship product');

[Using Sqoop Command, Load table to HDFS]

1.in node1/node2/node3

#load table from mysql to HDFS

$ sqoop import --connect jdbc:mysql://IP_Address_Of_Node1or2or3/hadoopguide --table mytable --username root --password 123456

#check on HDFS

$ hadoop fs -ls

[Some Useful Mysql Command]

show databases;
use databaseName;
show tables;
show columns from tableName; #查询表结构
select user(); #查询当前用户
select database(); #查询当前使用的数据库

[Trouble Shotting]

1.If Map-Reduce service is not start:

13/04/21 16:03:54 ERROR security.UserGroupInformation: PriviledgedActionException as:root (auth:SIMPLE) cause:java.net.ConnectException: Call to Ltwqzf25l/192.168.99.47:54311 failed on connection exception: java.net.ConnectException: Connection refused
13/04/21 16:03:54 ERROR tool.ImportTool: Encountered IOException running import job: java.net.ConnectException: Call to Ltwqzf25l/192.168.99.47:54311 failed on connection exception: java.net.ConnectException: Connection refused
at org.apache.hadoop.ipc.Client.wrapException(Client.java:1177)
at org.apache.hadoop.ipc.Client.call(Client.java:1153)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:225)
at org.apache.hadoop.mapred.$Proxy1.getProtocolVersion(Unknown Source)
at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:396)
at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:379)
at org.apache.hadoop.mapred.JobClient.createRPCProxy(JobClient.java:522)
at org.apache.hadoop.mapred.JobClient.init(JobClient.java:507)
at org.apache.hadoop.mapred.JobClient.<init>(JobClient.java:490)