sqoop部署

一、下载安装包

官网路径：

http://archive.apache.org/dist/sqoop/1.4.6/

下载压缩包：

sqoop-1.4.6.bin__hadoop-2.0.4-alpha.tar.gz

二、上传压缩包并解压

注意开启hadoop集群服务

[root@Hadoop01 software]# rz
[root@Hadoop01 software]# ll
总用量 607300
-rw-r--r--. 1 root root  92834839 2月   7 21:30 apache-hive-1.2.1-bin.tar.gz
-rw-r--r--. 1 root root 434354462 2月   2 21:16 hadoop-2.6.0-cdh5.16.2.tar.gz
drwxr-xr-x. 3 root root       171 2月  12 19:40 mysql-libs
-rw-r--r--. 1 root root  77807942 2月   7 21:44 mysql-libs.zip
-rw-r--r--. 1 root root  16870735 3月  23 20:45 sqoop-1.4.6.bin__hadoop-2.0.4-alpha.tar.gz

[root@Hadoop01 software]# tar -xzvf sqoop-1.4.6.bin__hadoop-2.0.4-alpha.tar.gz -C ../app
[root@Hadoop01 app]# ll
总用量 4
drwxr-xr-x.  8 root root  159 2月   7 21:35 apache-hive-1.2.1-bin
lrwxrwxrwx.  1 root root   22 2月   5 21:53 hadoop -> hadoop-2.6.0-cdh5.16.2
drwxr-xr-x. 16 root root  264 2月   5 22:50 hadoop-2.6.0-cdh5.16.2
lrwxrwxrwx.  1 root root   22 2月   7 21:40 hive -> apache-hive-1.2.1-bin/
drwxr-xr-x.  9 root root 4096 4月  27 2015 sqoop-1.4.6.bin__hadoop-2.0.4-alpha

做软连接

[root@Hadoop01 app]# ln -s sqoop-1.4.6.bin__hadoop-2.0.4-alpha/ sqoop

三、配置环境变量

按如下配置sqoop的路径

[root@Hadoop01 app]# vim /etc/profile
#sqoop
export SQOOP_HOME=/opt/app/sqoop
export PATH=$SQOOP_HOME/bin:$PATH

生效环境变量

[root@Hadoop01 app]# source /etc/profile

四、修改配置文件

配置文件都在sqoop下的conf文件夹中
复制sqoop-env-template.sh文件并重命名

[root@Hadoop01 sqoop]# cd conf
[root@Hadoop01 conf]# cp sqoop-env-template.sh sqoop-env.sh
[root@Hadoop01 conf]# vim sqoop-env.sh

按照如下配置Hadoop、hive等的路径

#Set path to where bin/hadoop is available
#export HADOOP_COMMON_HOME=/opt/app/hadoop

#Set path to where hadoop-*-core.jar is available
#export HADOOP_MAPRED_HOME=/opt/app/hadoop

#Set the path to where bin/hive is available
#export HIVE_HOME=opt/app/hive

五、jdbc与hive配置文件拷贝

将相应jdbc驱动拷贝到sqoop/lib中，以保证数据库与sqoop之间可以正常连接（mysql为例）

[root@Hadoop01 sqoop]# cp opt/software/mysql-libs/mysql-connector-java-5.1.27/mysql-connector-java-5.1.27-bin.jar ./lib/

将hive-site.xml拷贝到sqoop/conf/中

[root@Hadoop01 sqoop]# cp /opt/app/hive/conf/hive-site.xml ./conf/

六、测试是否部署成功

连接mysql，测试sqoop是否部署成功
连接mysql时，要在sqoop命令后填写连接参数

bin/sqoop 命令 \
--connect jdbc:mysql://服务器名:MySQL端口/库名 \
--username mysql账号 \
--password mysql密码

查看sqoop命令帮助

[root@Hadoop01 sqoop]# bin/sqoop help
Available commands:
  codegen            Generate code to interact with database records
  create-hive-table  Import a table definition into Hive
  eval               Evaluate a SQL statement and display the results
  export             Export an HDFS directory to a database table
  help               List available commands
  import             Import a table from a database to HDFS
  import-all-tables  Import tables from a database to HDFS
  import-mainframe   Import datasets from a mainframe server to HDFS
  job                Work with saved jobs
  list-databases     List available databases on a server
  list-tables        List available tables in a database
  merge              Merge results of incremental imports
  metastore          Run a standalone Sqoop metastore
  version            Display version information

查看mysql中的库

[root@Hadoop01 sqoop]# bin/sqoop list-databases --connect jdbc:mysql://hadoop01:3306/ --username 你的mysql账号 --password 你的mysql密码
21/03/23 21:35:33 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6
21/03/23 21:35:33 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
21/03/23 21:35:34 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
information_schema
metastore
mysql
performance_schema
test

查看mysql中test库中的表

[root@Hadoop01 sqoop]# bin/sqoop list-tables jdbc:mysql://hadoop01:3306/test --username 你的MySQL账号 --password 你的MySQL密码
21/03/23 21:36:20 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6
21/03/23 21:36:20 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
21/03/23 21:36:20 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
company