在Centos7上进行Hive 3.1.2集群安装

最新推荐文章于 2024-06-01 08:34:47 发布

Bulut0907

最新推荐文章于 2024-06-01 08:34:47 发布

阅读量601

点赞数

分类专栏：数据库和数据湖文章标签： hive 3.1.2 集群安装 centos7 数据仓库

本文链接：https://blog.csdn.net/yy8623977/article/details/122555847

版权

数据库和数据湖专栏收录该内容

29 篇文章 0 订阅

订阅专栏

1. 依赖组件安装
2. 下载并解压
3. 配置conf
- 3.1 配置Hive运行环境: hive-env.sh
- 3.2 配置Hive相关参数: hive-site.xml
4. 下载mysql-connector-java-8.0.25.jar到lib目录
5. 配置环境变量
6. 初始化Mysql元数据
7. 解决数据库、表、分区、索引的注释中文乱码
8. 启动metastore和HiveServer2
9. beeline连接Hive进行测试

注意：本文安装的Hive3.1.2不是stable版本

1. 依赖组件安装

组件	参考文章
Java8	centos7同时安装java8和openJdk11、windows同时安装java8和openJdk11
mysql	centos7安装mysql8.0.25版本
hadoop	Centos7上Hadoop 3.3.1的高可用HA安装过程

2. 下载并解压

[root@hive1 ~]#
[root@hive1 ~]# wget --no-check-certificate https://dlcdn.apache.org/hive/hive-3.1.2/apache-hive-3.1.2-bin.tar.gz
[root@hive1 ~]#
[root@hive1 ~]# tar -zxvf apache-hive-3.1.2-bin.tar.gz
[root@hive1 ~]# cd apache-hive-3.1.2-bin

3. 配置conf

3.1 配置Hive运行环境: hive-env.sh

复制hive-env.sh

[root@hive1 apache-hive-3.1.2-bin]# cp conf/hive-env.sh.template conf/hive-env.sh

修改内容如下：

# 指定Hadoop目录
HADOOP_HOME=/root/hadoop-3.3.1
# 指定Hive配置文件目录
export HIVE_CONF_DIR=/root/apache-hive-3.1.2-bin/conf
# 指定Hive依赖包目录
export HIVE_AUX_JARS_PATH=/root/apache-hive-3.1.2-bin/lib

添加内容如下：

export JAVA_HOME=/root/jdk1.8.0_201

3.2 配置Hive相关参数: hive-site.xml

[root@hive1 apache-hive-3.1.2-bin]# touch conf/hive-site.xml

文件内容如下：

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<configuration>


    <property>
        <name>hive.metastore.warehouse.dir</name>
        <value>hdfs://nnha/user/hive/warehouse</value>
        <description>配置Hive数据储存在HDFS上的目录</description>
    </property>

    <property>
        <name>hive.exec.scratchdir</name>
        <value>hdfs://nnha/tmp/hive</value>
        <description>配置Hive在HDFS上的临时目录</description>
    </property>

    <property>
        <name>hive.metastore.local</name>
        <value>false</value>
        <description>指定Hive是否开启本地模式, 此处表示远程模式</description>
    </property>

    <property>
        <name>hive.metastore.uris</name>
        <value>thrift://hive1:9083</value>
        <description>指定Metastore服务地址</description>
    </property>

    <property>
        <name>hive.metastore.event.db.notification.api.auth</name>
        <value>false</value>
        <description>指定Hive的metastore是否开启database权限认证</description>
    </property>

    <property>
        <name>hive.server2.active.passive.ha.enable</name>
        <value>true</value>
        <description>指定Hive是否开启ActivePassiveHA高可用</description>
    </property>

    <property>
        <name>javax.jdo.option.ConnectionURL</name>
        <value>jdbc:mysql://hive3:3306/hive?createDatabaseIfNotExist=true&amp;useSSL=false&amp;allowPublicKeyRetrieval=true</value>
        <description>配置JDBC连接地址, allowPublicKeyRetrieval指定是否允许客户端从服务器自动请求公钥</description>
    </property>


    <property>
        <name>javax.jdo.option.ConnectionDriverName</name>
        <value>com.mysql.cj.jdbc.Driver</value>
        <description>配置JDBC驱动</description>
    </property>

    <property>
        <name>javax.jdo.option.ConnectionUserName</name>
        <value>root</value>
        <description>配置连接MySQL的用户名</description>
    </property>

    <property>
        <name>javax.jdo.option.ConnectionPassword</name>
        <value>Root_123</value>
        <description>配置连接MySQL的密码</description>
    </property>

    <property>
        <name>hive.cli.print.header</name>
        <value>true</value>
        <description>配置在命令行界面HiveCLI中显示表的列名</description>
    </property>

    <property>
        <name>hive.cli.print.current.db</name>
        <value>true</value>
        <description>配置在命令行界面HiveCLI中显示当前数据库名称</description>
    </property>


</configuration>

4. 下载mysql-connector-java-8.0.25.jar到lib目录

[root@hive1 apache-hive-3.1.2-bin]# 
[root@hive1 apache-hive-3.1.2-bin]# wget -P /root/apache-hive-3.1.2-bin/lib https://repo1.maven.org/maven2/mysql/mysql-connector-java/8.0.25/mysql-connector-java-8.0.25.jar

5. 配置环境变量

向/root/.bashrc添加如下内容

export HIVE_HOME=/root/apache-hive-3.1.2-bin
export PATH=$PATH:$HIVE_HOME/bin

执行source /root/.bashrc使环境变量生效

6. 初始化Mysql元数据

初始化Mysql元数据

[root@hive1 ~]# 
[root@hive1 ~]# schematool -initSchema -dbType mysql
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/root/apache-hive-3.1.2-bin/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/root/hadoop-3.3.1/share/hadoop/common/lib/slf4j-log4j12-1.7.30.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Metastore connection URL:	 jdbc:mysql://hive3:3306/hive?createDatabaseIfNotExist=true&useSSL=false
Metastore Connection Driver :	 com.mysql.cj.jdbc.Driver
Metastore connection User:	 root
Starting metastore schema initialization to 3.1.0
Initialization script hive-schema-3.1.0.mysql.sql
......省略部分......
Initialization script completed
schemaTool completed
[root@hive1 ~]#

7. 解决数据库、表、分区、索引的注释中文乱码

登录Hive的Mysql元数据库，将元数据表字段的latin1字符类型，改成utf8类型

修改数据库注释

alter table DBS modify column `DESC` varchar(4000) character set utf8 default null;

修改表注释和表字段注释

alter table TABLE_PARAMS modify column PARAM_VALUE mediumtext character set utf8;
alter table COLUMNS_V2 modify column COMMENT varchar(256) character set utf8 default null;

修改分区字段注释

alter table PARTITION_PARAMS modify column PARAM_VALUE varchar(4000) character set utf8 default null;
alter table PARTITION_KEYS modify column PKEY_COMMENT varchar(4000) character set utf8 default null;

修改索引注释

alter table INDEX_PARAMS modify column PARAM_VALUE varchar(4000) character set utf8 default null;

注意：对于已经创建的Hive表，还是会出现中文注释乱码。

8. 启动metastore和HiveServer2

HiveServer2的作用：允许远程不同编程语言的客户端，向Hive提交查询请求。其核心是基于Thrift，由Thrift负责Hive的查询服务

[root@hive1 ~]# 
[root@hive1 ~]# mkdir apache-hive-3.1.2-bin/logs
[root@hive1 ~]# 
[root@hive1 ~]# nohup hive --service metastore >> /root/apache-hive-3.1.2-bin/logs/metastore.log 2>&1 &
[1] 13700
[root@hive1 ~]# nohup hive --service hiveserver2 >> /root/apache-hive-3.1.2-bin/logs/hiveserver2.log 2>&1 &
[2] 13817
[root@hive1 ~]#

如果出现问题，可以查看/tmp/root/hive.log日志

9. beeline连接Hive进行测试

用户名和密码为Centos服务器的用户名和密码

[root@hive1 ~]# beeline -u jdbc:hive2://hive1:10000 -n root -p root123
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/root/apache-hive-3.1.2-bin/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/root/hadoop-3.3.1/share/hadoop/common/lib/slf4j-log4j12-1.7.30.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/root/apache-hive-3.1.2-bin/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/root/hadoop-3.3.1/share/hadoop/common/lib/slf4j-log4j12-1.7.30.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Connecting to jdbc:hive2://hive1:10000
Connected to: Apache Hive (version 3.1.2)
Driver: Hive JDBC (version 3.1.2)
Transaction isolation: TRANSACTION_REPEATABLE_READ
Beeline version 3.1.2 by Apache Hive
0: jdbc:hive2://hive1:10000>
0: jdbc:hive2://hive1:10000> show databases;
INFO  : Compiling command(queryId=root_20220208144518_4eeac460-1ae4-469d-9ce2-81bce6071fc9): show databases
INFO  : Concurrency mode is disabled, not creating a lock manager
INFO  : Semantic Analysis Completed (retrial = false)
INFO  : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:database_name, type:string, comment:from deserializer)], properties:null)
INFO  : Completed compiling command(queryId=root_20220208144518_4eeac460-1ae4-469d-9ce2-81bce6071fc9); Time taken: 1.455 seconds
INFO  : Concurrency mode is disabled, not creating a lock manager
INFO  : Executing command(queryId=root_20220208144518_4eeac460-1ae4-469d-9ce2-81bce6071fc9): show databases
INFO  : Starting task [Stage-0:DDL] in serial mode
INFO  : Completed executing command(queryId=root_20220208144518_4eeac460-1ae4-469d-9ce2-81bce6071fc9); Time taken: 0.112 seconds
INFO  : OK
INFO  : Concurrency mode is disabled, not creating a lock manager
+----------------+
| database_name  |
+----------------+
| default        |
+----------------+
1 row selected (2.414 seconds)
0: jdbc:hive2://hive1:10000> 
0: jdbc:hive2://hive1:10000> !exit
Closing: 0: jdbc:hive2://hive1:10000
[root@hive1 ~]#

Bulut0907

关注

0
点赞
踩
5

收藏

觉得还不错? 一键收藏
0
评论
在Centos7上进行Hive 3.1.2集群安装

3. 配置conf3.1 配置Hive运行环境: hive-env.sh复制hive-env.sh修改内容如下：添加内容如下：3.2 配置Hive相关参数: hive-site.xml文件内容如下：4. 下载mysql-connector-java-8.0.25.jar到lib目录5. 配置环境变量向/root/.bashrc添加如下内容执行使环境变量生效初始化Mysql元数据7. 解决数据库、表、分区、索引的注释中文乱码登录Hive的Mysql元数据库，将元数据表字段的lati
复制链接

扫一扫

专栏目录