sqoop入门到熟悉

sqoop是什么

用于在hadoop体系和传统体系之间传输数据

sqoop安装

`
1、上传、解压、更名

[root@my01 ~]# tar -zxvf sqoop-1.4.7.bin__hadoop-2.6.0.tar.gz -C /usr/local/
[root@my01 local]# mv sqoop-1.4.7.bin__hadoop-2.6.0/ sqoop

2、配置环境变量

[root@my01 local]# vim /etc/profile
................
#sqoop environment
export SQOOP_HOME=/usr/local/sqoop
export PATH=$SQOOP_HOME/bin:$PATH
[root@my01 local]# source /etc/profile

3、配置sqoop的环境脚本文件

[root@my01 conf]# cp sqoop-env-template.sh sqoop-env.sh
#Set path to where bin/hadoop is available
export HADOOP_COMMON_HOME=/usr/local/hadoop
#Set path to where hadoop-*-core.jar is available
export HADOOP_MAPRED_HOME=/usr/local/hadoop
#set the path to where bin/hbase is available
export HBASE_HOME=/usr/local/hbase
#Set the path to where bin/hive is available
export HIVE_HOME=/usr/local/hive
#Set the path for where zookeper config dir is
export ZOOCFGDIR=/usr/local/zookeeper/conf

4、导入mysql驱动到lib目录

mysql-connector-java-5.1.xxx.jar

常用命令

sqoop help

usage: sqoop COMMAND [ARGS]

Available commands:
  codegen            Generate code to interact with database records
  create-hive-table  Import a table definition into Hive
  eval               Evaluate a SQL statement and display the results
  export             Export an HDFS directory to a database table
  help               List available commands
  import             Import a table from a database to HDFS
  import-all-tables  Import tables from a database to HDFS
  import-mainframe   Import datasets from a mainframe server to HDFS
  job                Work with saved jobs
  list-databases     List available databases on a server
  list-tables        List available tables in a database
  merge              Merge results of incremental imports
  metastore          Run a standalone Sqoop metastore
  version            Display version information

用法

mysql数据导入到hive中

其中import表示导入
connect参数表示mysql的连接信息,nshop为数据库名
username,password为mysql的用户名和密码
table为需要从mysql导入hive的表格
hive-import表示导入到hive
hive-overwrite表示覆盖
hive-table 表示要导入哪个hive的表中

/usr/local/sqoop/bin/sqoop import \
--connect jdbc:mysql://192.168.xx.1:3306/nshop \
--username root \
--password 123456 \
--table customer_attention \
--hive-import \
--hive-overwrite \
--hive-table ods_nshop_2003.ods_02_customer_attention \
-m 1