【sqoop】Linux环境下sqoop-1.4.7安装

最新推荐文章于 2024-07-22 23:32:12 发布

debimeng

最新推荐文章于 2024-07-22 23:32:12 发布

阅读量476

点赞数

分类专栏：大数据文章标签： sqoop-1.4.7安装 linux下sqoop安装 sqoop1

本文链接：https://blog.csdn.net/debimeng/article/details/102084940

版权

大数据专栏收录该内容

23 篇文章 1 订阅

订阅专栏

sqoop安装

前提条件：已安装好hadoop+hive+mysql(或其他数据库，这使用mysql)

一、安装

下载和解压
http://ftp.wayne.edu/apache/sqoop/
# tar -zxvf sqoop-1.4.7-bin-hadoop200.tar.gz -C /opt
# mv /opt/sqoop-1.4.7-bin-hadoop200 /opt/sqoop

修改配置文件
cp /opt/sqoop/conf/sqoop-env-template.sh /opt/sqoop/conf/sqoop-env.sh
vi /opt/sqoop/conf/sqoop-env.sh
#Set path to where bin/hadoop is available
export HADOOP_COMMON_HOME=/opt/hadoop3

#Set path to where hadoop-*-core.jar is available
export HADOOP_MAPRED_HOME=/opt/hadoop3

#set the path to where bin/hbase is available
#export HBASE_HOME=

#Set the path to where bin/hive is available
export HIVE_HOME=/opt/hive3

#Set the path for where zookeper config dir is
#export ZOOCFGDIR=

加入mysql的jdbc驱动包，因之前安装的hive下有这个包，直接从hive的Lib下拷贝到sqoop/lib下即可
cp mysql-connector-java-5.1.46-bin.jar /opt/sqoop/lib/

修改环境变量
vi /etc/profile
##add
export SQOOP_HOME=/opt/sqoop
export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:$HIVE_HOME/lib/*
export PATH=$PATH:$JAVA_HOME/bin/:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$MYSQL_HOME/bin:$HIVE_HOME/bin:$SQOOP_HOME/bin/

source /etc/profile

启动sqoop
$ sqoop

二、遇到的问题及解决方法
---------------------------1
hadoop 3.x Could not find or load main class org.apache.hadoop.mapreduce.v2.app.MRAppMaster

场景：

解决：
$HADOOP_HOME/etc/mapred-site.xml,增加以下配置

<property>
   <name>yarn.app.mapreduce.am.env</name>
   <value>HADOOP_MAPRED_HOME=${HADOOP_HOME}</value>
</property>
<property>
   <name>mapreduce.map.env</name>
   <value>HADOOP_MAPRED_HOME=${HADOOP_HOME}</value>
</property>
<property>
   <name>mapreduce.reduce.env</name>
   <value>HADOOP_MAPRED_HOME=${HADOOP_HOME}</value>
</property>

---------------------------2
ERROR tool.ImportTool: Import failed: Import job failed!

解决：<property>
<name>mapreduce.map.memory.mb</name>
<value>100</value>
</property>
<property>
<name>mapreduce.map.java.opts</name>
<value>-Xmx300M</value>
</property>
<property>
<name>mapreduce.reduce.memory.mb</name>
<value>100</value>
</property>
<property>
<name>mapreduce.reduce.java.opts</name>
<value>-Xmx300M</value>
</property>

根据虚拟机的内存设置

---------------------------3
ERROR hive.HiveConfig: Could not load org.apache.hadoop.hive.conf.HiveConf. Make sure HIVE_CONF_DIR is set correctly.
ERROR tool.ImportTool: Import failed: java.io.IOException: java.lang.ClassNotFoundException: org.apache.hadoop.hive.conf.HiveConf

解决：
在配置文件/etc/profile中添加环境变量
export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:$HIVE_HOME/lib/*