简单介绍
Sqoop是一个用来将Hadoop(Hive、HBase)和关系型数据库中的数据相互转移的工具,可以将一个关系型数据库(例如:MySQL ,Oracle ,Postgres等)中的数据导入到Hadoop的HDFS中,也可以将HDFS的数据导入到关系型数据库中。
下载解压
下载所需版本的Sqoop ,这里我下载的官网的Sqoop 。下载地址为:https://mirror.bit.edu.cn/apache/sqoop/1.4.7
tar -zxvf sqoop-1.4.7.bin__hadoop-2.6.0.tar.gz
配置环境变量
vim /etc/profile
添加环境变量:
export SQOOP_HOME=/usr/app/sqoop-1.4.6-cdh5.15.2
export PATH=$SQOOP_HOME/bin:$PATH
使得配置的环境变量立即生效:
source /etc/profile
修改配置
进入安装目录下的conf/
目录,拷贝Sqoop的环境配置模板sqoop-env.sh.template
/opt/app/sqoop-1.4.7.bin__hadoop-2.6.0/conf
cp sqoop-env-template.sh sqoop-env.sh
修改sqoop-env.sh,内容如下
加入数据库驱动
将MySQL驱动包拷贝到Sqoop安装目录的lib目录下,没有驱动的话可以用我这个 https://pan.baidu.com/s/17XoE_akSfnTOCgxwXSZC3w 提取码: khes
/opt/app/sqoop-1.4.7.bin__hadoop-2.6.0/lib
mysql-connector-java-5.1.39.jar(驱动)
验证
由于之前已经将sqoop的bin目录配置到环境变量,在任意目录下
sqoop version
出现对应的版本信息则代表配置成功:
但是明显可以看出两个Warning
警告是因为我们本身就没有用到HCatalog
和Accumulo
,忽略即可。Sqoop在启动时会去检查环境变量中是否有配置这些软件,如果想去除这些警告,可以修改bin/configure-sqoop
,注释掉不必要的检查。
vim /opt/app/sqoop-1.4.7.bin__hadoop-2.6.0/bin/configure-sqoop
注释掉这些不必要的检查。
# Check: If we can't find our dependencies, give up here.
if [ ! -d "${HADOOP_COMMON_HOME}" ]; then
echo "Error: $HADOOP_COMMON_HOME does not exist!"
echo 'Please set $HADOOP_COMMON_HOME to the root of your Hadoop installation.'
exit 1
fi
if [ ! -d "${HADOOP_MAPRED_HOME}" ]; then
echo "Error: $HADOOP_MAPRED_HOME does not exist!"
echo 'Please set $HADOOP_MAPRED_HOME to the root of your Hadoop MapReduce installation.'
exit 1
fi
## Moved to be a runtime check in sqoop.
if [ ! -d "${HBASE_HOME}" ]; then
echo "Warning: $HBASE_HOME does not exist! HBase imports will fail."
echo 'Please set $HBASE_HOME to the root of your HBase installation.'
fi
## Moved to be a runtime check in sqoop.
if [ ! -d "${HCAT_HOME}" ]; then
echo "Warning: $HCAT_HOME does not exist! HCatalog jobs will fail."
echo 'Please set $HCAT_HOME to the root of your HCatalog installation.'
fi
if [ ! -d "${ACCUMULO_HOME}" ]; then
echo "Warning: $ACCUMULO_HOME does not exist! Accumulo imports will fail."
echo 'Please set $ACCUMULO_HOME to the root of your Accumulo installation.'
fi
if [ ! -d "${ZOOKEEPER_HOME}" ]; then
echo "Warning: $ZOOKEEPER_HOME does not exist! Accumulo imports will fail."
echo 'Please set $ZOOKEEPER_HOME to the root of your Zookeeper installation.'
fi