说明
Sqoop是一个客户端软件,安装在任意机器上,不需要安装在Hadoop集群中。
安装过程
安装
cd /opt/hadoop/client/
wget http://mirror.bit.edu.cn/apache/sqoop/1.4.5/sqoop-1.4.5.bin__hadoop-2.0.4-alpha.tar.gz
tar -zxvf sqoop-1.4.5.bin__hadoop-2.0.4-alpha.tar.gz
mv sqoop-1.4.5.bin__hadoop-2.0.4-alpha sqoop
环境变量
在/etc/profile中增加环境变量:
1
2exportSQOOP_HOME=/opt/hadoop/client/sqoop
exportPATH=$SQOOP_HOME/bin:$PATH
配置文件:
创建配置文件:
1cp sqoop-env-template.sh sqoop-env.sh
修改配置文件:
1
2
3
4vi sqoop-env.sh
exportHADOOP_COMMON_HOME=/opt/hadoop/client/hadoop-2.4.1
exportHADOOP_MAPRED_HOME=/opt/hadoop/client/hadoop-2.4.1
exportHIVE_HOME=/opt/hadoop/client/hive
MySQL JDBC Jar包
将MySQL的JDBC Jar包拷贝到Sqoop的lib目录下:
1cp ~/mysql-connector-java-5.1.22-bin.jar /opt/hadoop/client/sqoop/lib/
运行
将MySQL的数据库szhouse中的house_project表导入HDFS中:
sqoop import --connect jdbc:mysql://10.255.1.6/LemonHouse --username root --password 123456 --table house_project
默认设置下导入到HDFS上的路径是: /user/{username}/{tablename}/(files),比如我的当前用户是hadoop,那么实际路径即: /user/hadoop/house_project/(files)。
[hadoop@ctrl ~]$ hdfs dfs -ls /user/hadoop/house_project
Found 5 items
-rw-r--r-- 3 hadoop supergroup 0 2015-02-27 16:00 /user/hadoop/house_project/_SUCCESS
-rw-r--r-- 3 hadoop supergroup 7972 2015-02-27 16:00 /user/hadoop/house_project/part-m-00000
-rw-r--r-- 3 hadoop supergroup 11074 2015-02-27 16:00 /user/hadoop/house_project/part-m-00001
-rw-r--r-- 3 hadoop supergroup 13917 2015-02-27 16:00 /user/hadoop/house_project/part-m-00002
-rw-r--r-- 3 hadoop supergroup 14020 2015-02-27 16:00 /user/hadoop/house_project/part-m-00003
如果要自定义路径需要增加参数:–warehouse-dir 比如:
sqoop import --connect jdbc:mysql://10.255.1.6/LemonHouse --username root --password 123456 --table house_project --warehouse-dir /tmp/data