sqoop-介绍以及安装

简介

sqoop是一款开源工具，用于在hive和传统数据库间mysql做数据传递，也就是可以从hive导入到关系型数据库也可以从关系型数据库导入到hive这意思

底层原理

事实上sqoop就是把底层原理翻译成了mapreduce程序
翻译出的mapreduce程序主要是对inputformat和outputformat进行定制

安装

前提：有java，有hadoop。

下载地址

http://mirrors.hust.edu.cn/apache/sqoop/1.4.6/

部署

解压
tar -zxf sqoop-1.4.6.bin__hadoop-2.0.4-alpha.tar.gz -C /opt/module/
然后修改配置文件
进入conf目录
mv sqoop-env-template.sh sqoop-env.sh
然后修改sqoop-env.sh
下面这几个有啥加啥

export HADOOP_COMMON_HOME=/opt/module/hadoop-2.7.2
export HADOOP_MAPRED_HOME=/opt/module/hadoop-2.7.2
export HIVE_HOME=/opt/module/hive
export ZOOKEEPER_HOME=/opt/module/zookeeper-3.4.10
export ZOOCFGDIR=/opt/module/zookeeper-3.4.10
export HBASE_HOME=/opt/module/hbase

然后为了连接mysql，要把驱动放进flume的lib目录下
就是mysql解压后的下面这个驱动

然后放进去

测试

1.看看配置成功了么
[root@node1 sqoop]# bin/sqoop help
Warning: /opt/module/sqoop/bin/…/…/hbase does not exist! HBase imports will fail.
Please set $HBASE_HOME to the root of your HBase installation.
Warning: /opt/module/sqoop/bin/…/…/hcatalog does not exist! HCatalog jobs will fail.
Please set $HCAT_HOME to the root of your HCatalog installation.
Warning: /opt/module/sqoop/bin/…/…/accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
20/07/29 18:14:15 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6
usage: sqoop COMMAND [ARGS]

Available commands:
codegen Generate code to interact with database records
create-hive-table Import a table definition into Hive
eval Evaluate a SQL statement and display the results
export Export an HDFS directory to a database table
help List available commands
import Import a table from a database to HDFS
import-all-tables Import tables from a database to HDFS
import-mainframe Import datasets from a mainframe server to HDFS
job Work with saved jobs
list-databases List available databases on a server
list-tables List available tables in a database
merge Merge results of incremental imports
metastore Run a standalone Sqoop metastore
version Display version information
有一堆的waring+命令蹦出来就成功了
2.测试连mysql
bin/sqoop list-databases --connect jdbc:mysql://localhost:3306/ --username root --password 123456

这样就成功了

hive导入到mysql

hive导入到RDBMS对于sqoop来说是export的过程，mysql导入到hive那么就是imlort的过程
例子：

sqoop export --connect jdbc:mysql://localhost:3306/export_data --username root --password 123456 --table student --num-mappers 1 --export-dir /user/hive/warehouse/student --input-fields-terminated-by "\t"