七、Sqoop组件的安装与配置（数据传送工具）

七、Sqoop组件的安装与配置（数据传送工具）

1、解压sqoop

解压sqoop安装包

[root@master ~]# tar zxvf sqoop-1.4.7.bin__hadoop-2.6.0.tar.gz -C /usr/local/src/

重命名sqoop文件夹

[root@master ~]# mv /usr/local/src/sqoop-1.4.7.bin__hadoop-2.6.0/ /usr/local/src/sqoop

2、配置sqoop环境

创建sqoop的配置文件sqoop-env.sh

复制sqoop-env-template.sh模板，并将模板重命名为sqoop-env.sh

[root@master conf]# cd /usr/local/src/sqoop/conf/
[root@master conf]# cp sqoop-env-template.sh sqoop-env.sh

修改sqoop-env.sh文件，添加Hadoop，HBase，hive等组件的安装路径

[root@master conf]# vim sqoop-env.sh

export HADOOP_COMMON_HOME=/usr/local/src/hadoop
export HADOOP_MAPRED_HOME=/usr/local/src/hadoop
export HBASE_HOME=/usr/local/src/hbase
export HIVE_HOME=/usr/local/src/hive

配置sqoop的环境变量

[root@master conf]# vim /etc/profile
export SQOOP_HOME=/usr/local/src/sqoop
export PATH=$PATH:$SQOOP_HOME/bin
export CLASSPATH=$CLASSPATH:$SQOOP_HOME/lib
[root@master conf]# source /etc/profile

把mysql驱动复制到sqoop的lib目录下

[root@master ~]# cp mysql-connector-java-5.1.47.jar /usr/local/src/sqoop/lib/

3.启动sqoop

启动Hadoop集群

[root@master ~]# su - hadoop
[hadoop@master ~]$ start-all.sh
[hadoop@master ~]$ jps
1286 NameNode
1897 Jps
1482 SecondaryNameNode
1630 ResourceManager

测试sqoop是否能够正常链接mysql

[hadoop@master ~]$ sqoop-list-databases --connect jdbc:mysql://localhost:3306/ --username root -P
Warning: /usr/local/src/sqoop/../hcatalog does not exist! HCatalog jobs will fail.
Please set $HCAT_HOME to the root of your HCatalog installation.
Warning: /usr/local/src/sqoop/../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
21/10/23 15:26:01 INFO sqoop.Sqoop: Running Sqoop version: 1.4.7
Enter password:
21/10/23 15:26:09 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
Sat Oct 23 15:26:09 CST 2021 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
information_schema
hive
mysql
performance_schema
sys

将hive驱动放到sqoop的lib目录下

[hadoop@master ~]$ cp /usr/local/src/hive/lib/hive-common-2.0.0.jar /usr/local/src/sqoop/lib/

利用sqoop将MySQL数据库中的表导入到hive中

sqoop import --connect jdbc:mysql://master:3306/sample --username root --password Password123$ --table book --fields-terminated-by '|' --delete-target-dir --num-mappers 1 --hive-import --hive-database sample --hive-table book

利用sqoop将MySQL数据库中的表导入到hdfs中

sqoop import --connect jdbc:mysql://master:3306/School --username root --password Password123$ --table student -m 1 -target-dir /user/hadoop/hive/exam

利用sqoop将MySQL数据库中的表新增的数据增量导入到hdfs中

sqoop import -connect jdbc:mysql://master:3306/School -username root -password Password123$ -table student -m 1 -target-dir /user/hadoop/hive/exam -check-column id -incremental append -last-value 3