Sqoop的安装和基本应用

本次大数据环境有三个节点，所扮演的角色如下：

hadoop01所扮演的角色如下：

[root@hadoop01 ~]# jps
18186 DataNode
18606 DFSZKFailoverController
5780 Jps
17905 QuorumPeerMain
18708 ResourceManager
18068 NameNode
18400 JournalNode
19241 HRegionServer
18833 NodeManager

hadoop02所扮演的角色如下：

[root@hadoop02 ~]# jps
23016 ResourceManager
22487 DataNode
1391 Jps
23120 HRegionServer
22576 JournalNode
22405 NameNode
22695 DFSZKFailoverController
22847 NodeManager
22339 QuorumPeerMain

hadoop03所扮演的角色如下：

[root@hadoop03 ~]# jps
19117 NodeManager
19011 JournalNode
18909 DataNode
19571 HRegionServer
18838 QuorumPeerMain
24682 Jps
19427 HMaster

基于这个集群安装Sqoop1.4.6

所使用的软件是：sqoop-1.4.6.bin__hadoop-2.0.4-alpha.tar.gz

          依赖包是：mysql-connector-java-5.1.32-bin.jar

一，安装

1，解压软件到指定的位置

[root@hadoop01 bigdata]# ll
total 1005240
-rw-r--r--. 1 root root  92834839 May 28 10:26 apache-hive-1.2.1-bin.tar.gz
-rw-r--r--. 1 root root 231535691 May 29 15:46 apache-hive-2.3.0-bin.tar.gz
-rw-r--r--. 1 root root 231740978 May 31 14:35 apache-hive-2.3.2-bin.tar.gz
-rw-r--r--. 1 root root  53092828 Jun  1 16:07 apache-tez-0.7.1-bin.tar.gz
-rw-r--r--. 1 root root 216929574 May 24 11:42 hadoop-2.7.5.tar.gz
-rw-r--r--. 1 root root  88957416 Jul  2 14:13 hbase-0.98.15-hadoop2-bin.tar.gz
-rw-r--r--. 1 root root    969020 May 28 10:26 mysql-connector-java-5.1.32-bin.jar
-rw-r--r--. 1 root root  16870735 May 28 17:12 sqoop-1.4.6.bin__hadoop-2.0.4-alpha.tar.gz
-rw-r--r--. 1 root root  29909691 May 29 10:54 sqoop-1.4.6-cdh5.5.2.tar.gz
-rw-r--r--. 1 root root   2197063 May 31 10:28 sqoop-1.4.6.tar.gz
-rw-r--r--. 1 root root  46600150 Jun  6 15:20 tez-0.8.3.tar.gz
-rw-r--r--. 1 root root  17699306 May 24 11:44 zookeeper-3.4.6.tar.gz
[root@hadoop01 bigdata]# pwd
/home/tools/bigdata
[root@hadoop01 bigdata]# tar -zxvf sqoop-1.4.6.bin__hadoop-2.0.4-alpha.tar.gz -C /home/bigdata/

2，配置环境变量

[root@hadoop01 ~]# cd /home/bigdata/
[root@hadoop01 bigdata]# mv sqoop-1.4.6.bin__hadoop-2.0.4-alpha sqoop-1.4.6

vi /etc/profile
export SQOOP_HOME=/home/bigdata/sqoop-1.4.6
export PATH=$PATH:$SQOOP_HOME/bin

source /etc/profile

 

3，配置文件的修改

 

[root@hadoop01 conf]# cp sqoop-env-template.sh sqoop-env.sh
[root@hadoop01 conf]# vi sqoop-env.sh 

# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

# included in all the hadoop scripts with source command
# should not be executable directly
# also should not be passed any arguments, since we need original $*

# Set Hadoop-specific environment variables here.

#Set path to where bin/hadoop is available
export HADOOP_COMMON_HOME=/home/bigdata/hadoop-2.7.5

#Set path to where hadoop-*-core.jar is available
export HADOOP_MAPRED_HOME=/home/bigdata/hadoop-2.7.5

#set the path to where bin/hbase is available
export HBASE_HOME=/home/bigdata/hbase-0.98

#Set the path to where bin/hive is available
export HIVE_HOME=/home/bigdata/hive-1.2.1

#Set the path for where zookeper config dir is
export ZOOCFGDIR=/home/bigdata/zookeeper-3.4.6

4，上传jar到sqoop的lib目录下

[root@hadoop01 ~]# cd /home/bigdata/sqoop-1.4.6/lib/
[root@hadoop01 lib]# cp /home/tools/bigdata/mysql-connector-java-5.1.32-bin.jar .

5，去警告

[root@hadoop01 sqoop-1.4.6]# vi bin/configure-sqoop
#if [ -z "${HBASE_HOME}" ]; then
#  if [ -d "/usr/lib/hbase" ]; then
#    HBASE_HOME=/usr/lib/hbase
#  else
#    HBASE_HOME=${SQOOP_HOME}/../hbase
#  fi
#fi
#if [ -z "${HCAT_HOME}" ]; then
#  if [ -d "/usr/lib/hive-hcatalog" ]; then
#    HCAT_HOME=/usr/lib/hive-hcatalog
#  elif [ -d "/usr/lib/hcatalog" ]; then
#    HCAT_HOME=/usr/lib/hcatalog
#  else
#    HCAT_HOME=${SQOOP_HOME}/../hive-hcatalog
#    if [ ! -d ${HCAT_HOME} ]; then
#       HCAT_HOME=${SQOOP_HOME}/../hcatalog
#    fi
#  fi
#fi
#if [ -z "${ACCUMULO_HOME}" ]; then
#  if [ -d "/usr/lib/accumulo" ]; then
#    ACCUMULO_HOME=/usr/lib/accumulo
#  else
#    ACCUMULO_HOME=${SQOOP_HOME}/../accumulo
#  fi
#fi


## Moved to be a runtime check in sqoop.
#if [ ! -d "${HBASE_HOME}" ]; then
#  echo "Warning: $HBASE_HOME does not exist! HBase imports will fail."
#  echo 'Please set $HBASE_HOME to the root of your HBase installation.'
#fi

## Moved to be a runtime check in sqoop.
#if [ ! -d "${HCAT_HOME}" ]; then
#  echo "Warning: $HCAT_HOME does not exist! HCatalog jobs will fail."
#  echo 'Please set $HCAT_HOME to the root of your HCatalog installation.'
#fi

#if [ ! -d "${ACCUMULO_HOME}" ]; then
#  echo "Warning: $ACCUMULO_HOME does not exist! Accumulo imports will fail."
#  echo 'Please set $ACCUMULO_HOME to the root of your Accumulo installation.'
#fi

二，基本应用（基于用户测试应用）

1，查看Sqoop的版本

[tuoming@hadoop01 ~]$ sqoop version
Warning: /home/bigdata/sqoop-1.4.6/../hcatalog does not exist! HCatalog jobs will fail.
Please set $HCAT_HOME to the root of your HCatalog installation.
Warning: /home/bigdata/sqoop-1.4.6/../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
18/07/03 08:27:43 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6
Sqoop 1.4.6
git commit id c0c5a81723759fa575844a0a1eae8f510fa32c25
Compiled by root on Mon Apr 27 14:38:36 CST 2015

2，测试是否能连到MySql

[tuoming@hadoop01 ~]$ sqoop list-databases --connect jdbc:mysql://hadoop01:3306/ --username root --password 000000
Warning: /home/bigdata/sqoop-1.4.6/../hcatalog does not exist! HCatalog jobs will fail.
Please set $HCAT_HOME to the root of your HCatalog installation.
Warning: /home/bigdata/sqoop-1.4.6/../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
18/07/03 08:29:13 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6
18/07/03 08:29:13 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
18/07/03 08:29:13 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
information_schema
hive_remote
mysql
oozie
performance_schema
sys
temp
test

 

3，MySql的数据导入Hive

 

sqoop import --connect jdbc:mysql://hadoop01:3306/test?characterEncoding=utf-8  --username root --password '000000'  -table student -hive-import -hive-table tuoming.temp_student -m 1 --fields-terminated-by ","

4，Hive的数据导入MySql

sqoop export --connect jdbc:mysql://hadoop01:3306/test  --username root --password 000000  --table  student --export-dir /user/hive_remote/warehouse/tuoming.db/temp_student --input-fields-terminated-by ','