sqoop搭建配置使用

1.下载文件，并解压
http://archive.apache.org/dist/sqoop/
下面是解压之后的：

2.文件配置
进入conf文件，将sqoop-env-template.sh命名为sqoop-env.sh，并修改sqoop-env.sh：

[root@hadoop001 software]# cat /opt/sqoop/conf/sqoop-env.sh
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

# included in all the hadoop scripts with source command
# should not be executable directly
# also should not be passed any arguments, since we need original $*

# Set Hadoop-specific environment variables here.

#Set path to where bin/hadoop is available
export HADOOP_COMMON_HOME=/opt/hadoop/hadoop-2.8.0/

#Set path to where hadoop-*-core.jar is available
export HADOOP_MAPRED_HOME=/opt/hadoop/hadoop-2.8.0/

#set the path to where bin/hbase is available
#export HBASE_HOME=

#Set the path to where bin/hive is available
export HIVE_HOME=/opt/hive/apache-hive-2.3.3-bin

#Set the path for where zookeper config dir is
export ZOOCFGDIR=/opt/zookeeper/

3.将mysql驱动包加入到sqoop下lib中

[root@hadoop001 software]# ll /opt/sqoop/lib/
  ...
-rw-r--r-- 1 root root 2428320 3月  29 13:20 mysql-connector-java-8.0.25.jar
  ...

4.配置环境变量：

[root@hadoop001 software]# cat ~/.bashrc
#Sqoop
export SQOOP_HOME=/opt/sqoop
export PATH=$PATH:$SQOOP_HOME/bin

  source ~/.bashrc

5.验证是否成功

 sqoop-version 或者 sqoop version

6.查看基本命令

sqoop help

7.对mysql简单操作
（1）查看mysql有哪些数据库

[root@hadoop001 software]# sqoop list-databases --connect jdbc:mysql://hadoop001:3306/  --username root --password 000000
Warning: /opt/sqoop/../hbase does not exist! HBase imports will fail.
Please set $HBASE_HOME to the root of your HBase installation.
Warning: /opt/sqoop/../hcatalog does not exist! HCatalog jobs will fail.
Please set $HCAT_HOME to the root of your HCatalog installation.
Warning: /opt/sqoop/../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
22/04/08 16:58:06 INFO sqoop.Sqoop: Running Sqoop version: 1.4.7
22/04/08 16:58:06 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
22/04/08 16:58:06 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
Loading class `com.mysql.jdbc.Driver'. This is deprecated. The new driver class is `com.mysql.cj.jdbc.Driver'. The driver is automatically registered via the SPI and manual loading of the driver class is generally unnecessary.
hive
information_schema
mysql
performance_schema

（2）查看mysql有哪些表

[root@hadoop001 software]# sqoop list-tables --connect jdbc:mysql://hadoop001:3306/mysql  --username root --password 000000
Warning: /opt/sqoop/../hbase does not exist! HBase imports will fail.
Please set $HBASE_HOME to the root of your HBase installation.
Warning: /opt/sqoop/../hcatalog does not exist! HCatalog jobs will fail.
Please set $HCAT_HOME to the root of your HCatalog installation.
Warning: /opt/sqoop/../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
22/04/08 17:01:04 INFO sqoop.Sqoop: Running Sqoop version: 1.4.7
22/04/08 17:01:04 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
22/04/08 17:01:04 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
Loading class `com.mysql.jdbc.Driver'. This is deprecated. The new driver class is `com.mysql.cj.jdbc.Driver'. The driver is automatically registered via the SPI and manual loading of the driver class is generally unnecessary.
columns_priv
db
event
func
general_log
help_category
help_keyword
help_relation
help_topic
host
ndb_binlog_index

8.sqoop数据导入
（1）将mysql数据导入到hdfs中（未指定hdfs地址，选择默认路径）

sqoop import --connect jdbc:mysql://hadoop001:3306/mysql --username root  --password 000000  --table user   -m 1

查看结果：

[root@hadoop001 software]# hadoop fs -ls /user/root/user
Found 2 items
-rw-r--r--   2 root supergroup          0 2022-04-08 17:26 /user/root/user/_SUCCESS
-rw-r--r--   2 root supergroup        631 2022-04-08 17:26 /user/root/user/part-m-00000
[root@hadoop001 software]# hadoop fs -cat /user/root/user/part-m-00000
localhost,root,*032197AE5731D4664921A6CCAC7CFCE6A0698693,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,,,,,0,0,0,0,,
hadoop001,root,*032197AE5731D4664921A6CCAC7CFCE6A0698693,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,,,,,0,0,0,0,,
127.0.0.1,root,*032197AE5731D4664921A6CCAC7CFCE6A0698693,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,,,,,0,0,0,0,,
::1,root,*032197AE5731D4664921A6CCAC7CFCE6A0698693,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,,,,,0,0,0,0,,
%,root,*032197AE5731D4664921A6CCAC7CFCE6A0698693,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,N,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,,,,,0,0,0,0,,

（2）将mysql数据导入到hdfs中（指定hdfs地址和分隔符）

sqoop import  --connect jdbc:mysql://hadoop001:3306/mysql  --username root --password root  --table help_keyword  --target-dir /user/root/user/user_table  --fields-terminated-by '&' -m 2

查看结果：

root@hadoop001 software]# hdfs dfs -cat /user/root/user/user_table/part-m-00000
0&JOIN
1&HOST
2&REPEAT
3&SERIALIZABLE
4&REPLACE
5&AT
6&SCHEDULE
7&RETURNS
8&STARTS
9&MASTER_SSL_CA
10&NCHAR
11&COLUMNS
12&COMPLETION
13&WORK
14&DATETIME