CentOS 7 安装Sqoop-1.4.7-bin-hadoop
前言
Sqoop是一个可以将数据在关系型数据库与Hadoop进行转移的工具,底层通过MapResult的MR任务将数据按批次的形式进行转移,同时自定义了对应的InputFormat和OutputFormat,吞吐量很大,功能很强。
下载地址:https://mirrors.bfsu.edu.cn/apache/sqoop/1.4.7/
1 下载安装包
# 1
tar -zxvf sqoop-1.4.7.bin__hadoop-2.6.0.tar.gz -C /opt/module
# 2
mv sqoop-1.4.7.bin__hadoop-2.6.0 sqoop
# 3
cd sqoop/
2 修改配置文件,加载驱动(以MySQL为例)
# 1
cp sqoop-env-template.sh sqoop-env.sh
# 2 vim sqoop-env.sh,添加以下信息
#Set path to where bin/hadoop is available
export HADOOP_COMMON_HOME=/opt/module/hadoop-2.7.7
#Set path to where hadoop-*-core.jar is available
export HADOOP_MAPRED_HOME=/opt/module/hadoop-2.7.7
#Set the path to where bin/hive is available
export HIVE_HOME=/opt/module/hive-1.2.2
#set the path to where bin/hbase is available
#export HBASE_HOME=
#Set the path for where zookeper config dir is
export ZOOKEEPER_HOME=/opt/module/zk-3.6.1
export ZOOCFGDIR=/opt/module/zk-3.6.1/conf
3 导入MySQL的连接驱动
# 拷贝驱动
cp /opt/software/mysql-connector-java-8.0.16.jar /opt/module/sqoop-1.4.7/lib/
#验证是否成功
bin/sqoop help
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
Warning: /opt/module/sqoop-1.4.7/../hcatalog does not exist! HCatalog jobs will fail.
Please set $HCAT_HOME to the root of your HCatalog installation.
Warning: /opt/module/sqoop-1.4.7/../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
Error: Could not find or load main class org.apache.hadoop.hbase.util.GetJavaProperty
21/01/22 12:59:20 INFO sqoop.Sqoop: Running Sqoop version: 1.4.7
usage: sqoop COMMAND [ARGS]
Available commands:
codegen Generate code to interact with database records
create-hive-table Import a table definition into Hive
eval Evaluate a SQL statement and display the results
export Export an HDFS directory to a database table
help List available commands
import Import a table from a database to HDFS
import-all-tables Import tables from a database to HDFS
import-mainframe Import datasets from a mainframe server to HDFS
job Work with saved jobs
list-databases List available databases on a server
list-tables List available tables in a database
merge Merge results of incremental imports
metastore Run a standalone Sqoop metastore
version Display version information