Centos7搭建Sqoop数据转移工具

全程hadoop/hadoop账号下操作

创建sqoop安装的目录

mkdir -p /home/hadoop/sqoop

解压到安装目录

tar  -zxvf  sqoop-1.4.7.bin__hadoop-2.6.0.tar.gz  -C  /home/hadoop/sqoop

把解压后生成的文件夹重新命名为sqoop-1.4.7

cd  /home/hadoop/sqoop

mv  sqoop-1.4.7.bin__hadoop-2.6.0  sqoop-1.4.7

添加如下的配置,并执行source刷新环境变量文件

vi  ~/.bash_profile

#  Sqoop config

export SQOOP_HOME=/home/hadoop/sqoop/sqoop-1.4.7

export PATH=$PATH:$SQOOP_HOME/bin

source  ~/.bash_profile

 

 

cd conf

 

cp  sqoop-env-template.sh  sqoop-env.sh

hadoop安装目录

export HADOOP_COMMON_HOME=/home/hadoop/hadoop-ha/hadoop/hadoop-2.8.5

export HADOOP_MAPRED_HOME=/home/hadoop/hadoop-ha/hadoop/hadoop-2.8.5

hbase安装目录

export HBASE_HOME=/home/hadoop/hbase/hbase-1.4.11

hive安装目录

export HIVE_HOME=/home/hadoop/hive/apache-hive-2.3.6-bin

zookeeper配置文件目录

export ZOOCFGDIR=/usr/local/zookeeper/zookeeper-3.4.14

修改configure-sqoop文件

cd /usr/local/services/sqoop/bin/

vi configure-sqoop

注释掉以下代码

#if [ -z "${HCAT_HOME}" ]; then

#  if [ -d "/usr/lib/hive-hcatalog" ]; then

#    HCAT_HOME=/usr/lib/hive-hcatalog

#  elif [ -d "/usr/lib/hcatalog" ]; then

#    HCAT_HOME=/usr/lib/hcatalog

#  else

#    HCAT_HOME=${SQOOP_HOME}/../hive-hcatalog

#    if [ ! -d ${HCAT_HOME} ]; then

#       HCAT_HOME=${SQOOP_HOME}/../hcatalog

#    fi

#  fi

#fi

#if [ -z "${ACCUMULO_HOME}" ]; then

#  if [ -d "/usr/lib/accumulo" ]; then

#    ACCUMULO_HOME=/usr/lib/accumulo

#  else

#    ACCUMULO_HOME=${SQOOP_HOME}/../accumulo

#  fi

#fi

## Moved to be a runtime check in sqoop.

#if [ ! -d "${HCAT_HOME}" ]; then

#  echo "Warning: $HCAT_HOME does not exist! HCatalog jobs will fail."

#  echo 'Please set $HCAT_HOME to the root of your HCatalog installation.'

#fi

#if [ ! -d "${ACCUMULO_HOME}" ]; then

#  echo "Warning: $ACCUMULO_HOME does not exist! Accumulo imports will fail."

#  echo 'Please set $ACCUMULO_HOME to the root of your Accumulo installation.'

#fi

mysql-connector-java-5.1.47.jar 拷贝该jar包到sqoop的lib目录

检验安装是否成功:

$ sqoop version
Warning: /home/hadoop/sqoop/sqoop-1.4.7/../zookeeper does not exist! Accumulo imports will fail.
Please set $ZOOKEEPER_HOME to the root of your Zookeeper installation.
20/03/24 17:28:22 INFO sqoop.Sqoop: Running Sqoop version: 1.4.7
Sqoop 1.4.7
git commit id 2328971411f57f0cb683dfb79d19d4d19d185dd8
Compiled by maugli on Thu Dec 21 15:59:58 STD 2017
 

测试是否安装成功：

sqoop list-databases --connect jdbc:mysql://ip:3306/hive_db --username root  --password 123456

看到下面的相关信息,说明配置成功

information_schema

hive_db

mysql

performance_schema

test