Centos7.x 搭建部署 Hive 3.1.2单台+集群

1.基础环境Hadoop3.2.2集群

mysql数据库
基于centos部署hadoop集群

2.Hive 的安装

1.下载hive安装文件

wget https://mirrors.bfsu.edu.cn/apache/hive/hive-3.1.2/apache-hive-3.1.2-bin.tar.gz

2.下载后进行解压 tar -zxf
3.进行配置文件修改，进入conf文件夹

cp hive-default.xml.template hive-site.xml
cp hive-env.sh.template hive-env.sh
cp hive-log4j2.properties.template hive-log4j2.properties

4.修改hive-site.xml，一下mysqlip需替换，以及密码信息

<configuration>
    <property>
        <name>hive.metastore.schema.verification</name>
        <value>false</value>
    </property>
    <property>
        <name>hive.metastore.warehouse.dir</name>
        <value>/data/hive/warehouse</value>
    </property>
    <property>
        <name>hive.exec.mode.local.auto</name>
        <value>false</value>
        <description> Let Hive determine whether to run in local mode automatically </description>
    </property>
    <property>
        <name>javax.jdo.option.ConnectionURL</name>
        <value>jdbc:mysql://mysqlip:3306/hive??useUnicode=true&amp;characterEncoding=utf8&amp;useSSL=false&amp;allowMultiQueries=true</value>
    </property>
    <property>
        <name>javax.jdo.option.ConnectionDriverName</name>
        <value>com.mysql.jdbc.Driver</value>
    </property>
    <property>
        <name>javax.jdo.option.ConnectionUserName</name>
        <value>root</value>
    </property>
    <property>
        <name>javax.jdo.option.ConnectionPassword</name>
        <value>root</value>
    </property>
    <!-- 显示表的列名 -->
    <property>
      <name>hive.cli.print.header</name>
      <value>true</value>
    </property>
    <!-- 显示数据库名称 -->
    <property>
      <name>hive.cli.print.current.db</name>
      <value>true</value>
    </property>
    <property>
      <name>hive.exec.local.scratchdir</name>
      <value>/data/apache-hive-3.1.2/tmp/</value>
    <description>Local scratch space for Hive jobs</description>
    </property>
    <property>
      <name>hive.downloaded.resources.dir</name>
      <value>/data/apache-hive-3.1.2/tmp/</value>
      <description>Temporary local directory for added resources in the remote file system.</description>
    </property>
</configuration>

5.修改hive-env.sh

​export HADOOP_HEAPSIZE=1024
HADOOP_HOME=/data/hadoop-3.2.2
export HIVE_CONF_DIR=/data/apache-hive-3.1.2/conf

6.修改hive-log4j2.properties

property.hive.log.dir = /data/apache-hive-3.1.2/logs

7.编辑bin目录下的hive-config.sh

export JAVA_HOME=/usr/java/java-1.8.0
export HADOOP_HOME=/data/hadoop-3.2.2
export HIVE_HOME=/data/apache-hive-3.1.2

8.修改环境变量

export HADOOP_HOME="/data/hadoop-3.2.2"
export HIVE_HOME="/data/apache-hive-3.1.2"
 
export HADOOP_HDFS_HOME=$HADOOP_HOME
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib:$HADOOP_HOME/lib/native"
 
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$HIVE_HOME/bin
export CLASSPATH=$HADOOP_HOME/share/hadoop/common/*:$HADOOP_HOME/share/hadoop/common/lib/*:$hive_HOME/lib

9.下载mysql驱动包放到lib目录下

wget https://repo1.maven.org/maven2/mysql/mysql-connector-java/5.1.48/mysql-connector-java-5.1.48.jar

10.检查guava版本是否一直，不一致则统一替换为高版本，不然初始化sql会报错
Exception in thread “main” java.lang.NoSuchMethodError: com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;Ljava/lang/Object;)
jar包目录
/apache-hive-3.1.2-bin/lib/
hadoop/share/hadoop/common/lib/
这里将hive的guava的jar包删除，从hadoop的进行复制

11.执行数据库初始化命令

schematool -initSchema -dbType mysql

Initialization script completed
schemaTool completed
表示初始化成功，打开数据库，可以看到相关表信息。

12.启动hive，执行命令

hive

三.基础命令

1.查看所有数据库列表

show databases;

2.创建数据库

create database db_test;

3.查看创建的数据库在 HDFS 中的信息（随意找一个 HDFS 客户端节点）

hadoop fs -ls /data/hive/warehouse
hdfs dfs -ls /user/hive/warehouse

4.创建表

use db_test;
create table t_test(id string,name string,age int);
insert into table_test values ('1000','名称',21);
select * from table_test;

四.集群部署

1.在hive-site.xml添加以下配置

  <property>
    <name>hive.server2.active.passive.ha.enable</name>
    <value>true</value>
    <description>Whether HiveServer2 Active/Passive High Availability be enabled when Hive Interactive sessions are enabled.This will also require hive.server2.support.dynamic.service.discovery to be enabled.</description>
  </property>

2.拷贝hive安装包到其他服务器

scp -r apache-hive-3.1.2 hadoop2:/data
scp -r apache-hive-3.1.2 hadoop3:/data

3.将其他两台的环境变量保持和当前一直
4.hiveserver2 服务启动

[root@master ~]# nohup hive --service hiveserver2 --hiveconf hive.server2.thrift.port=10001 &

5.访问控制台
http://hadoop1:10002/