本篇主要演示在Linux上基于Hadoop安装Sqoop1。
一 安装环境
- 操作系统:Oracle Linux Server release 6.5;
- Java版本:java-1.7.0-openjdk-1.7.0.45;
- Hadoop版本:hadoop-2.7.6;
- Sqoop2版本:1.4.7;
二 安装前准备
1 安装Hadoop
具体安装过程参考:
Hadoop 2.7.6安装_伪分布式集群
2 下载Sqoop1以及MySQL-jdbc
Sqoop1下载地址:
http://mirror.bit.edu.cn/apache/sqoop/1.4.7/
三 安装配置Sqoop1
1 解压缩安装包
[root@strong ~]# tar zxvf sqoop-1.4.7.bin__hadoop-2.6.0.tar.gz
[root@strong ~]# mv sqoop-1.4.7.bin__hadoop-2.6.0 sqoop-1.4.7
[root@strong ~]# chown -R hadoop:hadoop sqoop-1.4.7
2 修改配置文件
[root@strong ~]# cp sqoop-env-template.sh sqoop-env.sh
[hadoop@strong conf]$ vim sqoop-env.sh
#Set path to where bin/hadoop is available
export HADOOP_COMMON_HOME=/usr/local/hadoop
#Set path to where hadoop-*-core.jar is available
export HADOOP_MAPRED_HOME=/usr/local/hadoop
#set the path to where bin/hbase is available
export HBASE_HOME=/usr/local/hbase
#Set the path to where bin/hive is available
export HIVE_HOME=/usr/local/apache-hive-2.3.3-bin
export JAVA_HOME=/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.45.x86_64
3 拷贝Mysql驱动至Lib目录
[hadoop@strong ~]$ cp /usr/local/apache-hive-2.3.3-bin/lib/mysql-connector-java-5.1.46.jar /usr/local/sqoop-1.4.7/lib/
4 查看版本信息
[hadoop@strong ~]$ sqoop version
Warning: /usr/local/sqoop-1.4.7/../hcatalog does not exist! HCatalog jobs will fail.
Please set $HCAT_HOME to the root of your HCatalog installation.
Warning: /usr/local/sqoop-1.4.7/../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
Warning: /usr/local/sqoop-1.4.7/../zookeeper does not exist! Accumulo imports will fail.
Please set $ZOOKEEPER_HOME to the root of your Zookeeper installation.
18/06/26 11:39:57 INFO sqoop.Sqoop: Running Sqoop version: 1.4.7
Sqoop 1.4.7
git commit id 2328971411f57f0cb683dfb79d19d4d19d185dd8
Compiled by maugli on Thu Dec 21 15:59:58 STD 2017
注:上面有几个警告,因为没有启用对应的组件,暂时忽略。
5 查看帮助信息
[hadoop@strong ~]$ sqoop help
Warning: /usr/local/sqoop-1.4.7/../hcatalog does not exist! HCatalog jobs will fail.
Please set $HCAT_HOME to the root of your HCatalog installation.
Warning: /usr/local/sqoop-1.4.7/../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
Warning: /usr/local/sqoop-1.4.7/../zookeeper does not exist! Accumulo imports will fail.
Please set $ZOOKEEPER_HOME to the root of your Zookeeper installation.
18/06/26 11:42:32 INFO sqoop.Sqoop: Running Sqoop version: 1.4.7
usage: sqoop COMMAND [ARGS]
Available commands:
codegen Generate code to interact with database records
create-hive-table Import a table definition into Hive
eval Evaluate a SQL statement and display the results
export Export an HDFS directory to a database table
help List available commands
import Import a table from a database to HDFS
import-all-tables Import tables from a database to HDFS
import-mainframe Import datasets from a mainframe server to HDFS
job Work with saved jobs
list-databases List available databases on a server
list-tables List available tables in a database
merge Merge results of incremental imports
metastore Run a standalone Sqoop metastore
version Display version information
See 'sqoop help COMMAND' for information on a specific command.
6 验证对数据库的访问
[hadoop@strong ~]$ sqoop list-databases --connect jdbc:mysql://strong.hadoop.com:3306 --username root --password root
Warning: /usr/local/sqoop-1.4.7/../hcatalog does not exist! HCatalog jobs will fail.
Please set $HCAT_HOME to the root of your HCatalog installation.
Warning: /usr/local/sqoop-1.4.7/../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
Warning: /usr/local/sqoop-1.4.7/../zookeeper does not exist! Accumulo imports will fail.
Please set $ZOOKEEPER_HOME to the root of your Zookeeper installation.
18/06/26 11:50:53 INFO sqoop.Sqoop: Running Sqoop version: 1.4.7
18/06/26 11:50:53 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
18/06/26 11:50:54 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
Tue Jun 26 11:50:54 CST 2018 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
mysql
information_schema
performance_schema
sys
hive
至此,Sqoop1环境安装成功,下篇将演示Sqoop1的功能。