基于飞腾平台的Sqoop的安装配置

飞腾开发者

已于 2024-09-09 16:42:07 修改

阅读量1.1k

点赞数 15

文章标签： sqoop 大数据数据库 linux 运维

于 2024-09-09 16:41:39 首次发布

本文链接：https://blog.csdn.net/developersupport/article/details/142062743

版权

【写在前面】飞腾开发者平台是基于飞腾自身强大的技术基础和开放能力，聚合行业内优秀资源而打造的。该平台覆盖了操作系统、算法、数据库、安全、平台工具、虚拟化、存储、网络、固件等多个前沿技术领域，包含了应用使能套件、软件仓库、软件支持、软件适配认证四大板块，旨在共享尖端技术，为开发者提供一个涵盖多领域的开发平台和工具套件。点击这里开始你的技术升级之旅吧

本文分享至飞腾开发者平台《飞腾平台Sqoop1.99.7移植与安装手册》

1 介绍

Sqoop（SQL–to–Hadoop）是一个用来将关系型数据库和Hadoop中的数据进行相互转移的工具，可以将一个关系型数据库(例如Mysql、Oracle)中的数据导入到Hadoop(例如HDFS、Hive、Hbase)中，也可以将Hadoop(例如HDFS Hive、Hbase)中的数据导入到关系型数据库(例如Mysql、Oracle)中。

本文主要介绍移植适配后的Sqoop1.99.7在飞腾平台的安装与部署过程。

2 环境要求

2.1 硬件要求

硬件要求如下表所示。

项目	说明
CPU	FT-2000+/64服务器
网络	无要求
存储	无要求
内存	无要求

2.2 操作系统要求

操作系统要求如下表所示。

项目	说明
CentOS	8
Kernel	4.18.0-193.el8.aarch64

2.3 软件要求

软件要求如下表所示。

项目	说明
Java	1.8.0_281
MySQL	5.7
Hadoop	3.3.0

3 安装与部署

3.1 程序部署

下载apache-sqoop

wget http://archive.apache.org/dist/sqoop/1.99.7/sqoop-1.99.7-bin-hadoop200.tar.gz
mv sqoop-1.99.7-bin-hadoop200.tar.gz /opt
cd /opt/
tar -zxvf sqoop-1.99.7-bin-hadoop200.tar.gz
mv sqoop-1.99.7-bin-hadoop200.tar.gz sqoop-1.99.7

3.2 程序配置

1）配置环境变量

编辑 /etc/profile 文件，添加以下内容:

export SQOOP_HOME=/opt/sqoop-1.99.7
export PATH=$PATH:$SQOOP _HOME/bin

2）配置日志信息

# vim /opt/sqoop-1.99.7/conf/sqoop.properties
#日志配置
BASEDIR=/opt/sqoop-1.99.7
LOGDIR=/opt/sqoop-1.99.7/logs

org.apache.sqoop.log4j.appender.file.File=/opt/sqoop-1.99.7/logs/sqoop.log
org.apache.sqoop.repository.sysprop.derby.stream.error.file=/opt/sqoop-1.99.7/logs/derbyrepo.log

# hadoop配置
org.apache.sqoop.submission.engine.mapreduce.configuration.directory=/opt/hadoop-3.3.0/etc/hadoop
#授权配置
org.apache.sqoop.security.authentication.type=SIMPLE
org.apache.sqoop.security.authentication.handler=org.apache.sqoop.security.authentication.SimpleAuthenticationHandler
org.apache.sqoop.security.authentication.anonymous=true

3）进入$ SQOOP_HOME目录，并新建 conf/hive-site.xml 文件添加以下配置

<property>
    <name>hadoop.proxyuser.sqoop2.hosts</name>
    <value>hadoop</value>
</property>
<property>
    <name>hadoop.proxyuser.sqoop2.groups</name>
    <value>hadoop</value>
</property>

4）上传Mysql驱动包到hive的lib目录

cp mysql-connector-java.jar /opt/sqoop-1.99.7/server/lib/

3.3 启动服务

1）验证服务

[root@master sqoop-1.99.7]# sqoop2-tool verify
Setting conf dir: /usr/local/apps/sqoop-1.99.7/bin/../conf
Sqoop home directory: /usr/local/apps/sqoop-1.99.7
Sqoop tool executor:
Version: 1.99.7
Revision: 435d5e61b922a32d7bce567fe5fb1a9c0d9b1bbb
Compiled on Tue Jul 19 16:08:27 PDT 2016 by abefine
Running tool: class org.apache.sqoop.tools.tool.VerifyTool
0 [main] INFO org.apache.sqoop.core.SqoopServer – Initializing Sqoop server.
41 [main] INFO org.apache.sqoop.core.PropertiesConfigurationProvider – Starting config file poller thread
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/local/apps/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/local/apps/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
Exception in thread “main” java.lang.NoSuchMethodError: com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;Ljava/lang/Object;)V
at org.apache.hadoop.conf.Configuration.set(Configuration.java:1357)
at org.apache.hadoop.conf.Configuration.set(Configuration.java:1338)
at org.apache.sqoop.security.authentication.SimpleAuthenticationHandler.secureLogin(SimpleAuthenticationHandler.java:37)
at org.apache.sqoop.security.AuthenticationManager.initialize(AuthenticationManager.java:98)
at org.apache.sqoop.core.SqoopServer.initialize(SqoopServer.java:57)
at org.apache.sqoop.tools.tool.VerifyTool.runTool(VerifyTool.java:36)
at org.apache.sqoop.tools.ToolRunner.main(ToolRunner.java:72)

以上报错，经过分析后发现是guava版本问题，qoop内部依赖guava版本与hadoop依赖的guava版本存在冲突与兼容性问题。

解决方案：

[root@master sqoop-1.99.7-bin-hadoop200]# find /opt/sqoop-1.99.7/ -name guava*.jar
./tools/lib/guava-11.0.2.jar
./shell/lib/guava-11.0.2.jar
./server/lib/guava-11.0.2.jar
[root@master sqoop-1.99.7-bin-hadoop200]#
[root@master sqoop-1.99.7-bin-hadoop200]
[root@master sqoop-1.99.7-bin-hadoop200]# ls /opt/hadoop-3.3.0/share/hadoop/common/lib/guava-27.0-jre.jar
/opt/hadoop-3.3.0/share/hadoop/common/lib/guava-27.0-jre.jar
用hadoop的版本文件替换 /opt/sqoop-1.99.7/ 目录下的所有guava文件

再次验证：

[hadoop@master sqoop-1.99.7]$ sqoop2-tool  verify
Setting conf dir: /opt/sqoop-1.99.7/bin/../conf
Sqoop home directory: /opt/sqoop-1.99.7
Sqoop tool executor:
    Version: 1.99.7
    Revision: 435d5e61b922a32d7bce567fe5fb1a9c0d9b1bbb
    Compiled on Tue Jul 19 16:08:27 PDT 2016 by abefine
Running tool: class org.apache.sqoop.tools.tool.VerifyTool
0    [main] INFO  org.apache.sqoop.core.SqoopServer  - Initializing Sqoop server.
13   [main] INFO  org.apache.sqoop.core.PropertiesConfigurationProvider  - Starting config file poller thread
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/hadoop-3.3.0/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/hive-3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
Verification was successful.
Tool class org.apache.sqoop.tools.tool.VerifyTool has finished correctly.

2）升级检测

# bin/sqoop2-tool upgrader

结果显示：

[hadoop@master sqoop-1.99.7]# bin/sqoop2-tool upgrade
Setting conf dir: /opt/sqoop-1.99.7/bin/../conf
Sqoop home directory: /opt/sqoop-1.99.7
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
Exception in thread "main" java.lang.ExceptionInInitializerError
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:348)
    at org.apache.sqoop.utils.ClassUtils.loadClassWithClassLoader(ClassUtils.java:106)
    at org.apache.sqoop.utils.ClassUtils.loadClass(ClassUtils.java:70)
    at org.apache.sqoop.repository.JdbcRepositoryProvider.initializeRepositoryHandler(JdbcRepositoryProvider.java:122)
    at org.apache.sqoop.repository.JdbcRepositoryProvider.initialize(JdbcRepositoryProvider.java:65)
    at org.apache.sqoop.repository.RepositoryManager.initialize(RepositoryManager.java:124)
    at org.apache.sqoop.tools.tool.UpgradeTool.runToolWithConfiguration(UpgradeTool.java:39)
    at org.apache.sqoop.tools.ConfiguredTool.runTool(ConfiguredTool.java:35)
    at org.apache.sqoop.tools.ToolRunner.main(ToolRunner.java:72)
Caused by: java.lang.SecurityException: sealing violation: package org.apache.derby.impl.jdbc.authentication is sealed
    at java.net.URLClassLoader.getAndVerifyPackage(URLClassLoader.java:400)
    at java.net.URLClassLoader.definePackageInternal(URLClassLoader.java:420)
    at java.net.URLClassLoader.defineClass(URLClassLoader.java:452)
    at java.net.URLClassLoader.access$100(URLClassLoader.java:74)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:369)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:355)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:351)

以上报错，经过分析后发现是guava版本问题，qoop内部依赖derby版本与hive版本依赖的derby版本存在一致性问题。

解决方案：

[root@master sqoop-1.99.7]# find . -name  derby*.jar
./server/lib/derby-10.8.2.2.jar
[root@master sqoop-1.99.7]# find /opt/hive-3.1.2/ -name  derby*.jar
LICENSE                  RELEASE_NOTES.txt        bin/                     conf/                    hcatalog/                lib/
NOTICE                   auxlib/                  binary-package-licenses/ examples/                jdbc/                    scripts/
[root@master sqoop-1.99.7]# find /opt/hive-3.1.2/lib/ -name  derby*.jar
/opt/hive-3.1.2/lib/derby-10.14.1.0.jar
 [root@master sqoop-1.99.7]# mv server/lib/derby-10.8.2.2.jar server/lib/derby-10.8.2.2.jar.old
[root@master sqoop-1.99.7]# cp /opt/hive-3.1.2/lib/derby-10.14.1.0.jar server/lib/

再次验证：

[hadoop@master sqoop-1.99.7]$ bin/sqoop2-tool upgrade
Setting conf dir: /opt/sqoop-1.99.7/bin/../conf
Sqoop home directory: /opt/sqoop-1.99.7
Sqoop tool executor:
    Version: 1.99.7
    Revision: 435d5e61b922a32d7bce567fe5fb1a9c0d9b1bbb
    Compiled on Tue Jul 19 16:08:27 PDT 2016 by abefine
Running tool: class org.apache.sqoop.tools.tool.UpgradeTool
0    [main] INFO  org.apache.sqoop.core.PropertiesConfigurationProvider  - Starting config file poller thread
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/hadoop-3.3.0/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/hive-3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
Tool class org.apache.sqoop.tools.tool.UpgradeTool has finished correctly.

3）启动服务

[hadoop@master sqoop-1.99.7]$ /opt/sqoop-1.99.7/bin/sqoop.sh server start
Setting conf dir: /opt/sqoop-1.99.7/bin/../conf
Sqoop home directory: /opt/sqoop-1.99.7
Starting the Sqoop2 server...
0    [main] INFO  org.apache.sqoop.core.SqoopServer  - Initializing Sqoop server.
9    [main] INFO  org.apache.sqoop.core.PropertiesConfigurationProvider  - Starting config file poller thread
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/hadoop-3.3.0/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/hive-3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
Sqoop2 server started.

检测端口服务：

[hadoop@master opt]$ sudo netstat -anpt|grep 1137694
tcp6    0   0 :::12000        :::*          LISTEN   
1137694/java

端口与服务均正常，表示sqoop启动正常。

4 功能测试

4.1 启动客户端测试

1）运行sqoop2客户端命令

[hadoop@master sqoop-1.99.7]$ sqoop2-shell
Setting conf dir: /opt/sqoop-1.99.7/bin/../conf
Sqoop home directory: /opt/sqoop-1.99.7
Sqoop Shell: Type 'help' or '\h' for help.

[ERROR] Failed to construct terminal; falling back to unsupported
java.lang.NumberFormatException: For input string: "0x100"
    at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
    at java.lang.Integer.parseInt(Integer.java:580)
    at java.lang.Integer.valueOf(Integer.java:766)
    at jline.internal.InfoCmp.parseInfoCmp(InfoCmp.java:59)
    at jline.UnixTerminal.parseInfoCmp(UnixTerminal.java:233)
    at jline.UnixTerminal.<init>(UnixTerminal.java:64)
    at jline.UnixTerminal.<init>(UnixTerminal.java:49)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
    at java.lang.Class.newInstance(Class.java:442)
    at jline.TerminalFactory.getFlavor(TerminalFactory.java:209)
    at jline.TerminalFactory.create(TerminalFactory.java:100)
    at jline.TerminalFactory.get(TerminalFactory.java:184)

启动客户端报错，解决方案如下：

[hadoop@master sqoop-1.99.7]$ export TERM=xterm-color

再次运行sqoop2客户端命令

[hadoop@master sqoop-1.99.7]$ $SQOOP_HOME/bin/sqoop.sh client
Setting conf dir: /opt/sqoop-1.99.7/bin/../conf
Sqoop home directory: /opt/sqoop-1.99.7
Sqoop Shell: Type 'help' or '\h' for help.
sqoop:000>

2）连接sqoop2-server

sqoop:000> set server --host localhost --port 12000 --webapp sqoop
Server is set successfully

3）查看当前sqoop2所有连接

sqoop:000> show connector
0    [main] WARN  org.apache.hadoop.util.NativeCodeLoader  - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
+------------------------+---------+------------------------------------------------------------+----------------------+
|          Name          | Version |                           Class                            | Supported Directions |
+------------------------+---------+------------------------------------------------------------+----------------------+
| generic-jdbc-connector | 1.99.7  | org.apache.sqoop.connector.jdbc.GenericJdbcConnector       | FROM/TO              |
| kite-connector         | 1.99.7  | org.apache.sqoop.connector.kite.KiteConnector              | FROM/TO              |
| oracle-jdbc-connector  | 1.99.7  | org.apache.sqoop.connector.jdbc.oracle.OracleJdbcConnector | FROM/TO              |
| ftp-connector          | 1.99.7  | org.apache.sqoop.connector.ftp.FtpConnector                | TO                   |
| hdfs-connector         | 1.99.7  | org.apache.sqoop.connector.hdfs.HdfsConnector              | FROM/TO              |
| kafka-connector        | 1.99.7  | org.apache.sqoop.connector.kafka.KafkaConnector            | TO                   |
| sftp-connector         | 1.99.7  | org.apache.sqoop.connector.sftp.SftpConnector              | TO                   |
+------------------------+---------+------------------------------------------------------------+----------------------+

结果表明：Sqoop2程序在飞腾平台下运行结果正确，符合预期，功能正常。

推荐阅读

欢迎广大开发者来飞腾开发者平台获取更多前沿技术文档及资料

如开发者在使用飞腾产品有任何问题可通过在线工单联系我们

版权所有。飞腾信息技术有限公司 2023。保留所有权利。

未经本公司同意，任何单位、公司或个人不得擅自复制，翻译，摘抄本文档内容的部分或全部，不得以任何方式或途径进行传播和宣传。

商标声明

Phytium和其他飞腾商标均为飞腾信息技术有限公司的商标。

本文档提及的其他所有商标或注册商标，由各自的所有人拥有。

注意

本文档的内容视为飞腾的保密信息，您应当严格遵守保密任务；未经飞腾事先书面同意，您不得向任何第三方披露本文档内容或提供给任何第三方使用。

由于产品版本升级或其他原因，本文档内容会不定期进行更新。除非另有约定，本文档仅作为使用指导，飞腾在现有技术的基础上尽最大努力提供相应的介绍及操作指引，但飞腾在此明确声明对本文档内容的准确性、完整性、适用性、可靠性的等不作任何明示或暗示的保证。

本文档中所有内容，包括但不限于图片、架构设计、页面布局、文字描述，均由飞腾和/或其关联公司依法拥有其知识产权，包括但不限于商标权、专利权、著作权等。非经飞腾和/或其关联公司书面同意，任何人不得擅自使用、修改，复制上述内容。