安装Hadoop组件Hive,配置Oracle存储元数据

终有一刻

已于 2024-07-29 19:50:54 修改

阅读量766

点赞数 23

于 2024-07-29 19:49:36 首次发布

本文链接：https://blog.csdn.net/weixin_42732268/article/details/140778281

版权

大数据专栏收录该内容

5 篇文章 0 订阅

订阅专栏

文章目录

1、准备工具和环境

1.1 hadoop环境(之前已经安装，在Centos7中安装Hadoop详细教程
)
1.2 hive-3.1.2 下载
地址：通过百度网盘分享的文件：apache-hive-3.1.2-bin.tar.gz
链接：https://pan.baidu.com/s/16_mmcJlpzNCYwZOqCwqlzQ
提取码：6ykd
1.3 oracle12C (之前安装过，无须再介绍）
1.4 ojdbc8 (该版本的驱动支持jdk1.8)
地址通过百度网盘分享的文件：ojdbc8.jar
链接：https://pan.baidu.com/s/1m5C_m8lFRBGXRxBEWqmIzw
提取码：iqbd

2、Hive安装

2.1 使用WinSCP 上传文件到linux系统，已下载的apache-hive-3.1.2-bin.tar.gz上传至/opt/tools/
2.2 复制到/opt/server/,并改名为hive-3.1.2

hang server]$ ls
flume  hadoop-2.8.0  hive-3.1.2  hue-3.9.0-cdh5.14.0

3、配置环境变量

3.1、使用vim编辑配置文件

 # vim /etc/profile

3.2、在配置文件末添加

export HIVE_HOME=/opt/server/hive-3.1.2
export PATH=$HIVE_HOME/bin:$PATH

3.3、使环境变量生效

 source /etc/profile

4、修改Hive配置

4.1 修改hive-env.sh
进入/opt/server/hive-3.1.2/conf
将hive-env.sh.template重命名为hive-env.sh，使用vim编辑
hive-env.sh,增加hadoop_home路径

# HADOOP_HOME=${bin}/../../hadoop
HADOOP_HOME=/opt/server/hadoop-2.8.0

4.2 添加hive-site.xml
新建hive-site.xml 文件，内容如下，主要是配置存放元数据的Oracle的URL、驱动、用户名和密码等信息：

 version="2.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
  <property>
    <name>javax.jdo.option.ConnectionURL</name>
    <value>jdbc:oracle:thin:@zhang:1521:ORCL</value>
  </property>

  <property>
    <name>javax.jdo.option.ConnectionDriverName</name>
    <value>oracle.jdbc.OracleDriver</value>
  </property>

  <property>
    <name>javax.jdo.option.ConnectionUserName</name>
    <value>c##_zhangr</value>
  </property>

  <property>
    <name>javax.jdo.option.ConnectionPassword</name>
    <value>zhang</value>
  </property>

<property>
      <name>hive.server2.enable.doAs</name>
      <value>false</value>
  </property>
</configuration>

连接用户名使用的是oracle公共用户名c##_zhangr

5、拷贝oracle驱动程序到hive的lib下

如下所示

/opt/server/hive-3.1.2/lib
[oracle@zhang lib]$ ls ojdbc8.jar 
ojdbc8.jar

6、初始化数据

使用的hive 是2.x+版本时，必须手动初始化元数据库，1.x版本的会自动初始化，可省略该步操作，这里的Hive版本是3.x，故需要做初始化。初始化命令：

schematool -dbType oracle -initSchema

7、启动Hive

7.1、首先进入/opt/server/hive/bin

[oracle@zhang hive-3.1.2]$ pwd
/opt/server/hive-3.1.2

7.2、开启服务

root@zhang bin]# ./hive --service metastore &
[1] 5165
[root@zhang bin]# 2024-07-30 02:08:43: Starting Hive Metastore Server
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/server/hive-3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/server/hadoop-2.8.0/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
[root@zhang bin]#

7.3、打开Hive命令行

[root@zhang bin]# ./hive
which: no hbase in (/opt/server/hive-3.1.2/bin:/opt/server/flume/bin:/usr/lib/jvm/jdk1.8.0_65/bin:/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/bin:/sbin:/usr/lib/jvm/jdk1.8.0_65/bin:/opt/server/hadoop-2.8.0/bin:/opt/server/hadoop-2.8.0/sbin:/home/zhang/.local/bin:/home/zhang/bin)
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/server/hive-3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/server/hadoop-2.8.0/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Hive Session ID = 0450a178-2daf-44bb-8347-497cadcc45c5

Logging initialized using configuration in jar:file:/opt/server/hive-3.1.2/lib/hive-common-3.1.2.jar!/hive-log4j2.properties Async: true
Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
Hive Session ID = 573afdaa-0534-4db4-acbb-5996e47a8ed8
hive>

7.4、验证:输入简单SQL语句

hive> show databases;
OK
default
userdb
Time taken: 0.915 seconds, Fetched: 2 row(s)
hive>

验证正常，可正常执行SQL语句
7.5、如要使用外部应用连接Hive,需要开启服务
service hiveserver2 &
代码如下

./hive --service hiveserver2 &
[1] 5411
[zhang@zhang bin]$ which: no hbase in (/opt/server/hive-3.1.2/bin:/opt/server/flume/bin:/usr/lib/jvm/jdk1.8.0_65/bin:/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/bin:/sbin:/usr/lib/jvm/jdk1.8.0_65/bin:/opt/server/hadoop-2.8.0/bin:/opt/server/hadoop-2.8.0/sbin:/home/zhang/.local/bin:/home/zhang/bin)
2024-07-30 02:09:15: Starting HiveServer2
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/server/hive-3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/server/hadoop-2.8.0/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Hive Session ID = 8fe5cc1a-82a5-4227-9b6b-6f85d3c2c0ed
Hive Session ID = f77969f0-78fa-4de1-a881-abc12454bc5d
Hive Session ID = 2eea9353-057d-48cd-9fe6-04880cd8b298
Hive Session ID = 99227d48-0ebe-4d32-8514-a30894cd9012

8、小结

8.1、出现问题
hive远程连接提示：
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException
User: root is not allowed to impersonate admin

解决方法
在 core-site.xml中加入

  <property>    
    <name>hadoop.proxyuser.root.hosts</name>    
    <value>*</value>
  </property> 
  <property>    
    <name>hadoop.proxyuser.root.groups</name>    
    <value>*</value> 
  </property>

在 hive-site中加入配置，重启 hive --service hivserver2

  <property>
      <name>hive.server2.enable.doAs</name>
      <value>false</value>
  </property>

8.2、忘记开启oracle监听，也会出现无法启动hive 服务，所以务必启动oracle监听

[oracle@zhang ~]$ lsnrctl start

LSNRCTL for Linux: Version 12.2.0.1.0 - Production on 30-JUL-2024 02:07:27

Copyright (c) 1991, 2016, Oracle.  All rights reserved.

Starting /orcl/app/oracle/product/12.2.0/db_home/bin/tnslsnr: please wait...

TNSLSNR for Linux: Version 12.2.0.1.0 - Production
System parameter file is /orcl/app/oracle/product/12.2.0/db_home/network/admin/listener.ora
Log messages written to /orcl/app/oracle/diag/tnslsnr/zhang/listener/alert/log.xml
Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=zhang)(PORT=1521)))
Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=EXTPROC1521)))

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=zhang)(PORT=1521)))
STATUS of the LISTENER
------------------------
Alias                     LISTENER
Version                   TNSLSNR for Linux: Version 12.2.0.1.0 - Production
Start Date                30-JUL-2024 02:07:29
Uptime                    0 days 0 hr. 0 min. 14 sec
Trace Level               off
Security                  ON: Local OS Authentication
SNMP                      OFF
Listener Parameter File   /orcl/app/oracle/product/12.2.0/db_home/network/admin/listener.ora
Listener Log File         /orcl/app/oracle/diag/tnslsnr/zhang/listener/alert/log.xml
Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=zhang)(PORT=1521)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=EXTPROC1521)))
The listener supports no services
The command completed successfully