CentOS 7 安装Sqoop-1.4.7-bin-hadoop

CentOS 7 安装Sqoop-1.4.7-bin-hadoop

前言

Sqoop是一个可以将数据在关系型数据库与Hadoop进行转移的工具，底层通过MapResult的MR任务将数据按批次的形式进行转移，同时自定义了对应的InputFormat和OutputFormat，吞吐量很大，功能很强。

下载地址：https://mirrors.bfsu.edu.cn/apache/sqoop/1.4.7/

1 下载安装包

# 1 
tar -zxvf sqoop-1.4.7.bin__hadoop-2.6.0.tar.gz -C /opt/module
# 2
mv sqoop-1.4.7.bin__hadoop-2.6.0  sqoop
# 3 
cd sqoop/

2 修改配置文件，加载驱动(以MySQL为例)

# 1
cp sqoop-env-template.sh  sqoop-env.sh 

# 2 vim sqoop-env.sh，添加以下信息
#Set path to where bin/hadoop is available
export HADOOP_COMMON_HOME=/opt/module/hadoop-2.7.7
#Set path to where hadoop-*-core.jar is available
export HADOOP_MAPRED_HOME=/opt/module/hadoop-2.7.7
#Set the path to where bin/hive is available
export HIVE_HOME=/opt/module/hive-1.2.2
#set the path to where bin/hbase is available
#export HBASE_HOME=
#Set the path for where zookeper config dir is
export ZOOKEEPER_HOME=/opt/module/zk-3.6.1
export ZOOCFGDIR=/opt/module/zk-3.6.1/conf

3 导入MySQL的连接驱动

# 拷贝驱动 
cp /opt/software/mysql-connector-java-8.0.16.jar /opt/module/sqoop-1.4.7/lib/

#验证是否成功
bin/sqoop help

>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
Warning: /opt/module/sqoop-1.4.7/../hcatalog does not exist! HCatalog jobs will fail.
Please set $HCAT_HOME to the root of your HCatalog installation.
Warning: /opt/module/sqoop-1.4.7/../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
Error: Could not find or load main class org.apache.hadoop.hbase.util.GetJavaProperty
21/01/22 12:59:20 INFO sqoop.Sqoop: Running Sqoop version: 1.4.7
usage: sqoop COMMAND [ARGS]

Available commands:
  codegen            Generate code to interact with database records
  create-hive-table  Import a table definition into Hive
  eval               Evaluate a SQL statement and display the results
  export             Export an HDFS directory to a database table
  help               List available commands
  import             Import a table from a database to HDFS
  import-all-tables  Import tables from a database to HDFS
  import-mainframe   Import datasets from a mainframe server to HDFS
  job                Work with saved jobs
  list-databases     List available databases on a server
  list-tables        List available tables in a database
  merge              Merge results of incremental imports
  metastore          Run a standalone Sqoop metastore
  version            Display version information