Linux sqoop安装教程

Sqoop环境搭建前提，已经安装Jdk,mysql,Hadoop,Hive,Zookeeper,HBase。

sqoop-1.4.6-cdh5.14.2.tar.gz下载
链接：https://pan.baidu.com/s/12ycO9ekiFAZji7yL1Jou5A
提取码：laj8

mysql 5.1.38驱动包下载
链接：https://pan.baidu.com/s/1fSkQYaqUmBSBCVT1rWCNMw
提取码：cqlu

java-json.jar 下载
链接：https://pan.baidu.com/s/1W4WyQzI-BeiofLRUaa5GyQ
提取码：j8bp

以下介绍伪分布式环境下sqoop安装，可实现mysql数据导入HDFS文件，HDFS 向mysql中导出数据，mysql导入数据到hive。

Sqoop安装

1.sqoop、mysql、java-json压缩包上传到Linux中/opt文件夹中，并解压压缩包，移动到指定文件夹/opt/soft中

tar -zxf /opt/sqoop-1.4.6-cdh5.14.2.tar.gz 
mv /opt/sqoop-1.4.6-cdh5.14.2 /opt/soft/sqoop146

2.配置环境变量,并激活配置文件

vi /etc/profile
----------------------------------------
export SQOOP_HOME=/opt/install/sqoop 
export PATH=$SQOOP_HOME/bin:$PATH

source /etc/profile

3.mysql驱动包和java-json.jar 复制到sqoop安装文件lib目录中。

cp /opt/mysql-connector-java-5.1.38.jar  /opt/soft/sqoop146/lib
cp /opt/java-json.jar  /opt/soft/sqoop146/lib

缺少mysql驱动包，会出现如下错误：

ERROR sqoop.Sqoop: Got exception running Sqoop: java.lang.RuntimeException: 
Could not load db driver class: com.mysql.jdbc.Driver

缺少java-json.jar，会出现如下错误：

Exception in thread "main" java.lang.NoClassDefFoundError: org/json/JSONObject

4.进入sqoop/conf目录，复制sqoop-env-template.sh 重命名为sqoop-en.sh，修改该文件。

cd /opt/soft/sqoop146/conf
cp sqoop-env-template.sh sqoop-en.sh
vi sqoop-en.sh
--------------------------------------------------------------------
 #hadoop安装路径、hive安装路径、zookeeper安装路径、zookeeper配置文件路径、hbase安装路径
 export HADOOP_COMMON_HOME=/opt/soft/hadoop260  
 export HADOOP_MAPRED_HOME=/opt/soft/hadoop260
 export HIVE_HOME=/opt/soft/hive110
 export ZOOKEEPER_HOME=/opt/soft/zookeeper345
 export ZOOCFGDIR=/opt/soft/zookeeper345/conf
 export HBASE_HOME=/opt/soft/hbase120

5.hive安装文件lib中复制hive-common-1.1.0-cdh5.14.2.jar和hive-shims*到sqoop安装文件lib目录中。

cp /opt/soft/hive110/lib/hive-common-1.1.0-cdh5.14.2.jar /opt/soft/sqoop146/lib/
cp /opt/soft/hive110/lib/hive-shims* /opt/soft/sqoop146/lib/

缺少hive-common包，会出现如下问题：

20/07/15 22:26:56 ERROR tool.ImportTool: Import failed: 
java.io.IOException: java.lang.ClassNotFoundException: org.apache.hadoop.hive.conf.HiveConf

缺少hive-shims类包，会出现如下问题：

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/hive/shims/ShimLoader

6.sqoop list-databases 可以查询到连接mysql数据库中的所有数据库,则sqoop安装完成。

sqoop list-databases --username root --password 1111 --connect jdbc:mysql://192.168.197.106:3306/
---------------------------------------------------------------------
Warning: /opt/soft/sqoop146/../hcatalog does not exist! HCatalog jobs will fail.
Please set $HCAT_HOME to the root of your HCatalog installation.
Warning: /opt/soft/sqoop146/../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
20/07/23 19:08:54 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6-cdh5.14.2
20/07/23 19:08:54 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
20/07/23 19:08:54 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/soft/hadoop260/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/soft/hbase120/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
information_schema
eat
hive
mysql
performance_schema
retail_db
sqoop