电商数仓项目(九) Sqoop安装与配置

---

本节讲解Sqoop 的安装和配置。
commons-lang-2.6.jar下载

一、规划

node01	node02	node03
sqoop	sqoop	sqoop

二、安装与配置

# 1. 下载sqoop
[jack@node01 u02]$ wget https://mirrors.tuna.tsinghua.edu.cn/apache/sqoop/1.4.7/sqoop-1.4.7.bin__hadoop-2.6.0.tar.gz

# 2. 解压缩并重命名
[jack@node01 u02]$ tar -zxf sqoop-1.4.7.bin__hadoop-2.6.0.tar.gz -C /u01
[jack@node01 u02]$ cd /u01
[jack@node01 u01]$ mv sqoop-1.4.7.bin__hadoop-2.6.0 sqoop-1.4.7

# 3. 修改环境变量,所有服务器都需要配置
[jack@node01 bin]$ sudo vi /etc/profile
export SQOOP_HOME=/u01/sqoop-1.4.7
export PATH=$PATH:$SQOOP_HOME/bin
[jack@node01 bin]$ source /etc/profile

# 4. 修改配置文件
[jack@node01 sqoop-1.4.7]$ cd conf
[jack@node01 conf]$ mv sqoop-env-template.sh sqoop-env.sh
[jack@node01 conf]$ vi sqoop-env.sh
export HADOOP_COMMON_HOME=/u01/hadoop-3.2.2
export HADOOP_MAPRED_HOME=/u01/hadoop-3.2.2
export ZOOKEEPER_HOME=/u01/zookeeper-3.5.8
export ZOOCFGDIR=/u01/zookeeper-3.5.8/conf

三、验证与测试

1. 下载mysql的jdbc：https://dev.mysql.com/downloads/connector/j/
   
2. 复制到sqoop的lib目录下

[jack@node01 u02]$ tar -zxf mysql-connector-java-8.0.22.tar.gz
[jack@node01 u02]$ cd mysql-connector-java-8.0.22
[jack@node01 mysql-connector-java-8.0.22]$ cp mysql-connector-java-8.0.22.jar /u01/sqoop-1.4.7/lib

3. 验证sqoop
   警告可以先忽略

[jack@node01 sqoop-1.4.7]$ bin/sqoop help
Warning: /u01/sqoop-1.4.7/bin/../../hbase does not exist! HBase imports will fail.
Please set $HBASE_HOME to the root of your HBase installation.
Warning: /u01/sqoop-1.4.7/bin/../../hcatalog does not exist! HCatalog jobs will fail.
Please set $HCAT_HOME to the root of your HCatalog installation.
Warning: /u01/sqoop-1.4.7/bin/../../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
2020-12-12 22:39:02,106 INFO sqoop.Sqoop: Running Sqoop version: 1.4.7
usage: sqoop COMMAND [ARGS]

Available commands:
  codegen            Generate code to interact with database records
      create-hive-table  Import a table definition into Hive

4. 测试sqoop和mysql连接
   报错处理：由于缺少commons-lang-2.6.jar，从https://mvnrepository.com/下载，上传到$SQOOP_HOME/lib目录下。(源码中有这个jar)

[jack@node01 sqoop-1.4.7]$ bin/sqoop list-databases --connect jdbc:mysql://node02:3306/ --username root -P

2020-12-12 22:46:13,949 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/commons/lang/StringUtils

5. Sqoop从mysql到hive时间少12小时
   在mysql中执行

show VARIABLES like "%time_zone%";

set global time_zone = '+08:00';
set time_zone = '+08:00';
flush priviledges;

6. 分发到其他服务器

[jack@node01 ~]$ cd /u01/bin
[jack@node01 bin]$ xsync /u01/sqoop-1.4.7

---

电商数仓项目(一) 系统规划和配置
电商数仓项目(二) Maven 安装和hadoop-lzo编译
电商数仓项目(三) hadoop3.2.2 安装与配置
电商数仓项目(四) 模拟电商日志数据开发
电商数仓项目(五) azkaban安装、配置和使用
电商数仓项目(六) zookeeper安装和配置
电商数仓项目(七) kafka 安装、配置和简单操作
电商数仓项目(八) Flume(1) 安装和配置
电商数仓项目(八) Flume(2) 拦截器开发
电商数仓项目(八) Flume(3) 生产者和消费者配置