大数据之Sqoop(一):Sqoop介绍与安装部署

记录sqoop的一些使用，复习下，此前项目中有用到，复习下

简介

• 用于hadoop(hive)与传统数据库Mysql等间进行数据传递，可让他们互相导入导出
• sqoop2与sqoop1不兼容

原理

• 导入导出命令翻译成mapreduce程序实现
• 在翻译出的mapreduce中主要对inputformat和outputformat进行定制

安装

准备

• java
• hadoop
• mysql

下载部署

• 下载地址：http://mirrors.hust.edu.cn/apache/sqoop/1.4.6/

• 上传安装包sqoop-1.4.6.bin__hadoop-2.0.4-alpha.tar.gz到虚拟机中

• 解压sqoop安装包到指定目录

 tar -zxf sqoop-1.4.6.bin__hadoop-2.0.4-alpha.tar.gz -C /home/mym/apps/

修改配置文件

Sqoop根目录下的conf目录中

sqoop-env.sh

[root@mym conf]# cp sqoop-env-template.sh  sqoop-env.sh
[root@mym conf]# vi sqoop-env.sh 

export HADOOP_COMMON_HOME=/home/mym/apps/hadoop-2.4.1
export HADOOP_MAPRED_HOME=/home/mym/apps/hadoop-2.4.1
export HIVE_HOME=/home/mym/apps/hive
export ZOOKEEPER_HOME=/usr/local/zookeeper
export ZOOCFGDIR=/usr/local/zookeeper
export HBASE_HOME=/home/mym/apps/hbas

拷贝jdbc驱动

拷贝jdbc的驱动到sqoop的lib目录下

cp mysql-connector-java-5.1.47.jar /home/mym/apps/sqoop/lib/

验证Sqoop

[root@mym sqoop]# bin/sqoop help
Warning: /home/mym/apps/sqoop/bin/../../hcatalog does not exist! HCatalog jobs will fail.
Please set $HCAT_HOME to the root of your HCatalog installation.
Warning: /home/mym/apps/sqoop/bin/../../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
19/12/26 10:43:17 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6
usage: sqoop COMMAND [ARGS]

Available commands:
  codegen            Generate code to interact with database records
  create-hive-table  Import a table definition into Hive
  eval               Evaluate a SQL statement and display the results
  export             Export an HDFS directory to a database table
  help               List available commands
  import             Import a table from a database to HDFS
  import-all-tables  Import tables from a database to HDFS
  import-mainframe   Import datasets from a mainframe server to HDFS
  job                Work with saved jobs
  list-databases     List available databases on a server
  list-tables        List available tables in a database
  merge              Merge results of incremental imports
  metastore          Run a standalone Sqoop metastore
  version            Display version information

See 'sqoop help COMMAND' for information on a specific command.

测试是否能成功连接数据库

[root@mym sqoop]# bin/sqoop list-databases --connect jdbc:mysql://mym:3306/ --username root --password 123456
Warning: /home/mym/apps/sqoop/bin/../../hcatalog does not exist! HCatalog jobs will fail.
Please set $HCAT_HOME to the root of your HCatalog installation.
Warning: /home/mym/apps/sqoop/bin/../../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
19/12/26 12:04:10 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6
19/12/26 12:04:10 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
19/12/26 12:04:11 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
information_schema
hive
mysql
performance_schema