Apache Sqoop （最新版本）

最新推荐文章于 2024-10-01 07:00:00 发布

麦田里的守望者·

最新推荐文章于 2024-10-01 07:00:00 发布

阅读量657

点赞数 1

分类专栏： Sqoop 工具

本文链接：https://blog.csdn.net/weixin_38231448/article/details/104219366

版权

Apache Sqoop是一个用于在Hadoop和关系数据库之间批量传输数据的工具。本文介绍了Sqoop的安装、配置，以及如何使用sqoop-import从HDFS导入数据到MySQL和HBASE，以及使用sqoop-export将数据导出回关系数据库。详细步骤包括设置环境变量、配置数据库连接、执行全表和增量导入、导出到Hive和HBase等。

摘要由CSDN通过智能技术生成

概述

Apache Sqoop（TM）是一种旨在在Apache Hadoop和结构化数据存储（例如关系数据库）之间高效传输批量数据的工具。通过内嵌的MapReduce程序实现关系型数据库和HDFS、Hbase、Hive等数据的倒入导出。
在这里插入图片描述

安装

1、访问sqoop的网址http://sqoop.apache.org/，选择相应的sqoop版本下载，本案例选择下载的是1.4.7，下载完相应的工具包后，解压Sqoop:

[root@CentOS ~]# tar -zxf sqoop-1.4.7.bin__hadoop-2.6.0.tar.gz -C /usr/
[root@CentOS ~]# cd /usr/
[root@CentOS usr]# mv sqoop-1.4.7.bin__hadoop-2.6.0 sqoop-1.4.7
[root@CentOS ~]# cd /usr/sqoop-1.4.7/

2、配置SQOOP_HOME环境变量

[root@CentOS sqoop-1.4.7]# vi ~/.bashrc 
M2_HOME=/usr/apache-maven-3.3.9
SQOOP_HOME=/usr/sqoop-1.4.7
HIVE_HOME=/usr/apache-hive-1.2.2-bin
ZOOKEEPER_HOME=/usr/zookeeper-3.4.6
JAVA_HOME=/usr/java/latest
HADOOP_HOME=/usr/hadoop-2.9.2
HBASE_HOME=/usr/hbase-1.2.4
PATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$HBASE_HOME/bin:$M2_HOME/bin:$HIVE_HOME/bin:$SQOOP_HOME/bin
CLASSPATH=.
export JAVA_HOME
export PATH
export CLASSPATH
export HADOOP_HOME
export HBASE_HOME
export M2_HOME
export HADOOP_CLASSPATH=`hbase classpath`:/root/mapreduce-1.0-SNAPSHOT.jar
export HIVE_HOME
export ZOOKEEPER_HOME
export SQOOP_HOME
[root@CentOS sqoop-1.4.7]# source ~/.bashrc

3、修改conf下的sqoop-env.sh.template配置文件

[root@CentOS7 sqoop-1.4.7]#  mv conf/sqoop-env-template.sh conf/sqoop-env.sh
[root@CentOS7 sqoop-1.4.7]# vi conf/sqoop-env.sh


# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

# included in all the hadoop scripts with source command
# should not be executable directly
# also should not be passed any arguments, since we need original $*

# Set Hadoop-specific environment variables here.

#Set path to wher

最低0.47元/天解锁文章