sqoop install

最新推荐文章于 2024-10-02 08:23:00 发布

四美

最新推荐文章于 2024-10-02 08:23:00 发布

阅读量113

点赞数

分类专栏： hadoop 文章标签： hadoop

本文链接：https://blog.csdn.net/yfm081616/article/details/119059548

版权

hadoop 专栏收录该内容

5 篇文章 0 订阅

订阅专栏

本文档详细记录了Apache Sqoop的安装步骤，包括下载、解压、配置环境变量、配置SQOOP、修改Hadoop的core-site.xml以及处理Sqoop与Hadoop版本冲突的问题。在初始化和验证过程中，由于guava.jar版本不一致导致的错误，通过替换guava.jar解决了问题。最后成功启动sqoop服务并进行了版本验证。

摘要由CSDN通过智能技术生成

下载地址：
https://archive.apache.org/dist/sqoop/
我这里下载的是
sqoop-1.99.7-bin-hadoop200.tar.gz

mkdir -p /data/sqoop将压缩包放到该目录

#解压
tar -zxvf sqoop-1.99.7-bin-hadoop200.tar.gz

#配置环境变量
vi /etc/profile
export SQOOP_HOME=/data/sqoop/sqoop-1.99.7-bin-hadoop200
export PATH=$SQOOP_HOME/bin:$PATH

source /etc/profile

#配置 SQOOP
vi $SQOOP_HOME/conf/sqoop.properties
1.新建 logs 目录/data/sqoop/logdir替换文件中的 @LOGDIR@
2.新建 metadb 目录/data/sqoop/metadb替换文件中的 @BASEDIR@
3.Hadoop configuration directory 改成 Hadoop 的配置文件的目录，$HADOOP_HOME/etc/hadoop

#修改 HADOOP 的core-site.xml 
<property>
		<name>hadoop.proxyuser.root.hosts</name>
		<value>*</value>
</property>
<property>
		<name>hadoop.proxyuser.root.groups</name>
		<value>*</value>
</property>

将 mysql 的驱动放到 sqoop 的安装目录下的 server 下的 lib 目录下

#初始化并验证
./bin/sqoop2-tool upgrade
./bin/sqoop2-tool verify
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
Exception in thread "main" java.lang.NoSuchMethodError: com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;Ljava/lang/Object;)V
	at org.apache.hadoop.conf.Configuration.set(Configuration.java:1357)
	at org.apache.hadoop.conf.Configuration.set(Configuration.java:1338)
	at org.apache.sqoop.security.authentication.SimpleAuthenticationHandler.secureLogin(SimpleAuthenticationHandler.java:37)
	at org.apache.sqoop.security.AuthenticationManager.initialize(AuthenticationManager.java:98)
	at org.apache.sqoop.core.SqoopServer.initialize(SqoopServer.java:57)
	at org.apache.sqoop.tools.tool.VerifyTool.runTool(VerifyTool.java:36)
	at org.apache.sqoop.tools.ToolRunner.main(ToolRunner.java:72)


[root@yfm19 sqoop-1.99.7-bin-hadoop200]# pwd
/data/sqoop/sqoop-1.99.7-bin-hadoop200
[root@yfm19 sqoop-1.99.7-bin-hadoop200]# ll server/lib/ | grep guava
-rw-r--r-- 1 root root  1648200 Jul 20  2016 guava-11.0.2.jar

sqoop内依赖的guava.jar和hadoop内的版本不一致

[root@yfm19 sqoop-1.99.7-bin-hadoop200]# find -name *guava*
./server/lib/guava-27.0-jre.jar
./server/lib/guava-11.0.2.jar-bak
./shell/lib/guava-27.0-jre.jar
./shell/lib/guava-11.0.2.jar-bak
./tools/lib/guava-27.0-jre.jar
./tools/lib/guava-11.0.2.jar-bak


cp /data/hadoop-pretend/hadoop-3.2.2/share/hadoop/common/lib/guava-27.0-jre.jar ./server/lib/
mv server/lib/guava-11.0.2.jar server/lib/guava-11.0.2.jar-bak

#运行
./bin/sqoop2-server start
./bin/sqoop2-shell

sqoop:000> show version --all
client version:
  Sqoop 1.99.7 source revision 435d5e61b922a32d7bce567fe5fb1a9c0d9b1bbb 
  Compiled by abefine on Tue Jul 19 16:08:27 PDT 2016
0    [main] WARN  org.apache.hadoop.util.NativeCodeLoader  - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
server version:
  Sqoop 1.99.7 source revision 435d5e61b922a32d7bce567fe5fb1a9c0d9b1bbb 
  Compiled by abefine on Tue Jul 19 16:08:27 PDT 2016
API versions:
  [v1]