Apache Sqoop介绍&安装

概述

Apache Sqoop（TM）是一种旨在在Apache Hadoop和结构化数据存储（例如关系数据库）之间高效传输批量数据的工具。通过内嵌的MapReduce程序实现关系型数据库和HDFS、Hbase、Hive等数据的倒入导出。

安装

1、访问sqoop的网址http://sqoop.apache.org/，选择相应的sqoop版本下载，本案例选择下载的是1.4.7下载地址：https://mirrors.tuna.tsinghua.edu.cn/apache/sqoop/1.4.7/sqoop-1.4.7.bin__hadoop-2.6.0.tar.gz，下载完相应的工具包后，解压Sqoop

[root@CentOS ~]# tar -zxf sqoop-1.4.7.bin__hadoop-2.6.0.tar.gz -C /usr/
[root@CentOS ~]# cd /usr/
[root@CentOS usr]# mv sqoop-1.4.7.bin__hadoop-2.6.0 sqoop-1.4.7
[root@CentOS ~]# cd /usr/sqoop-1.4.7/

2、配置SQOOP_HOME环境变量

[root@CentOS sqoop-1.4.7]# vi ~/.bashrc 
JAVA_HOME=/usr/java/latest
HADOOP_HOME=/usr/hadoop-2.9.2
HBASE_HOME=/usr/hbase-1.2.4
HIVE_HOME=/usr/apache-hive-1.2.2-bin
SQOOP_HOME=/usr/sqoop-1.4.7
ZOOKEEPER_HOME=/usr/zookeeper-3.4.6
PATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$HBASE_HOME/bin:$HIVE_HOME/bin:$SQOOP_HOME/bin:$ZOOKEEPER_HOME/bin
CLASSPATH=.
export JAVA_HOME
export HADOOP_HOME
export HBASE_HOME
export HIVE_HOME
export PATH
export CLASSPATH
export SQOOP_HOME
export ZOOKEEPER_HOME
[root@CentOS sqoop-1.4.7]# source ~/.bashrc

3、修改conf下的sqoop-env.sh.template配置文件

[root@CentOS sqoop-1.4.7]# mv conf/sqoop-env-template.sh conf/sqoop-env.sh 
[root@CentOS sqoop-1.4.7]# vi conf/sqoop-env.sh 
#Set path to where bin/hadoop is available
export HADOOP_COMMON_HOME=/usr/hadoop-2.9.2

#Set path to where hadoop-*-core.jar is available
export HADOOP_MAPRED_HOME=/usr/hadoop-2.9.2

#set the path to where bin/hbase is available
export HBASE_HOME=/usr/hbase-1.2.4

#Set the path to where bin/hive is available
export HIVE_HOME=/usr/apache-hive-1.2.2-bin

#Set the path for where zookeper config dir is
export ZOOCFGDIR=/usr/zookeeper-3.4.6/conf

4、将MySQL驱动jar拷贝到sqoop的lib目录下

[root@CentOS ~]# cp /usr/apache-hive-1.2.2-bin/lib/mysql-connector-java-5.1.48.jar /usr/sqoop-1.4.7/lib/

5、验证Sqoop是否安装成功

[root@CentOS sqoop-1.4.7]# sqoop version
Warning: /usr/sqoop-1.4.7/../hbase does not exist! HBase imports will fail.
Please set $HBASE_HOME to the root of your HBase installation.
Warning: /usr/sqoop-1.4.7/../hcatalog does not exist! HCatalog jobs will fail.
Please set $HCAT_HOME to the root of your HCatalog installation.
Warning: /usr/sqoop-1.4.7/../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
Warning: /usr/sqoop-1.4.7/../zookeeper does not exist! Accumulo imports will fail.
Please set $ZOOKEEPER_HOME to the root of your Zookeeper installation.
19/12/22 08:40:12 INFO sqoop.Sqoop: Running Sqoop version: 1.4.7
Sqoop 1.4.7
git commit id 2328971411f57f0cb683dfb79d19d4d19d185dd8
Compiled by maugli on Thu Dec 21 15:59:58 STD 2017
[root@CentOS sqoop-1.4.7]# sqoop list-tables --connect jdbc:mysql://192.168.52.1:3306/mysql --username root --password root