Hive2.3.6伪分布式安装部署

Hive2.3.6伪分布式安装部署

一、Hive介绍

Hive依赖于HDFS存储数据，Hive将HQL转换成MapReduce执行，所以说Hive是基于Hadoop的一个数据仓库工具，实质就是一款基于HDFS的MapReduce计算框架，对存储在HDFS中的数据进行分析和管理。

二、环境依赖

Hive不能独立运行，需要依赖于一个RDBMS(存储元数据)和一个文件系统

安装环境：Centos7.6 + Hive 2.3.6

Hadoop伪分布式安装

三、Mysql安装

1、安装YUM Repo

1.1、由于CentOS的yum源中没有mysql,需要到mysql的官网中下载yum repo配置文件。

下载命令：

[root@master ~]# wget https://dev.mysql.com/get/mysql57-community-release-el7-9.noarch.rpm

1.2、然后进行repo的安装

[root@master ~]# rpm -ivh mysql57-community-release-el7-9.noarch.rpm
#执行完成后会在/etc/yum.repos.d/目录下生成两个repo文件mysql-community.repo mysql-community-source.repo
[root@master yum.repos.d]# ll
-rw-r--r--  1 root root 1416 Sep 12  2016 mysql-community.repo
-rw-r--r--  1 root root 1440 Sep 12  2016 mysql-community-source.repo

2、使用yum命令即可完成安装

注意：必须进入到/etc/yum.repos.d/目录后再执行以下脚本

2.1 安装命令

[root@master yum.repos.d]# yum install mysql-server

2.2 启动Mysql

[root@master yum.repos.d]# systemctl start mysqld   #启动

2.3 获取安装时的临时密码(在第一次登录时就是用这个密码)

[root@master yum.repos.d]# grep 'temporary password' /var/log/mysqld.log
2021-04-16T09:53:20.694030Z 1 [Note] A temporary password is generated for root@localhost: SEyAzLS5m/eX

2.4 若没有获取到临时密码，则

2.4.1 删除原来安装过的mysql残留的数据

[root@master yum.repos.d]# rm -rf /var/lib/mysql

2.4.2 再启动mysql

[root@master yum.repos.d]# systemctl start mysqld   #启动

3、登录验证

[root@master yum.repos.d]# mysql -u root -p 
Enter password: 
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 2
Server version: 5.7.33

Copyright (c) 2000, 2021, Oracle and/or its affiliates.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql>

#密码为临时密码

4、参数配置

4.1、修改密码

alter user 'root'@'localhost' identified by 'Abcd@123456';
#需要注意密码的规则，不然显示
#ERROR 1819 (HY000): Your password does not satisfy the current policy requirements

4.2、开启远程控制

#MySQL默认是没有开启远程控制的，必须添加远程访问的用户，即默认是只能自己访问，别的机器是访问不了的。
#连接服务器：
[root@master yum.repos.d]# mysql -u root -p 
#查看当前所有数据库
mysql> show databases;
#进入mysql数据库
mysql> use mysql
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed

#查看mysql数据库中所有的表：
mysql> show tables;

#查看user表中的数据：
mysql> select host,user from user;
+-----------+---------------+
| host      | user          |
+-----------+---------------+
| localhost | mysql.session |
| localhost | mysql.sys     |
| localhost | root          |
+-----------+---------------+
3 rows in set (0.00 sec)

#修改user表中的Host：
mysql> update user set Host='%' where User='root';
Query OK, 1 row affected (0.00 sec)
Rows matched: 1  Changed: 1  Warnings: 0
#说明：%代表任意的客户端，可替换成具体IP地址。

#最后刷新一下：
flush privileges;
#查看验证数据：
mysql> select host,user from user;
+-----------+---------------+
| host      | user          |
+-----------+---------------+
| %         | root          |
| localhost | mysql.session |
| localhost | mysql.sys     |
+-----------+---------------+
3 rows in set (0.00 sec)

#注意：一定要记得在写sql的时候要在语句完成后加上“;”

4.3、其他安全配置

#关闭MySQL
systemctl stop mysqld
#重启MySQL
systemctl restart mysqld
#查看MySQL运行状态
systemctl status mysqld
#设置开机启动
systemctl enable mysqld
#关闭开机启动
systemctl disable mysqld

#查看版本
mysql> select version();
+-----------+
| version() |
+-----------+
| 5.7.33    |
+-----------+
1 row in set (0.00 sec)

四、Hive安装

1、上传安装文件并解压缩

#使用之前的hadoop用户安装，并使用此用户进行上传
[hadoop@master apache-hive-2.3.6]$ tar -zxvf apache-hive-2.3.6-bin.tar.gz

2、配置Hive环境变量

执行命令

[hadoop@master ~]$ vim .bash_profile
#添加配置如下
export HADOOP_HOME=/home/hadoop/apps/hadoop-2.7.7
export HIVE_HOME=/home/hadoop/apps/apache-hive-2.3.6

PATH=$PATH:$HOME/.local/bin:$HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$HIVE_HOME/bin

export PATH
#使配置立即生效
[hadoop@master ~]$ source .bash_profile
#查看Hive的目录
[hadoop@master ~]$ echo $HIVE_HOME
/home/hadoop/apps/apache-hive-2.3.6

3、配置Hive元数据管理库

创建hive用户，密码hive，并创建hive源数据库

#创建数据库hive
mysql> create database hive;
Query OK, 1 row affected (0.00 sec)
#创建用户
mysql> create user 'hive'@'%' identified by 'Hive@123456';
#赋予权限
mysql> grant all on hive.* to 'hive'@'%';
Query OK, 0 rows affected (0.00 sec)

4、修改配置文件

[hadoop@master conf]$ cd /home/hadoop/apps/apache-hive-2.3.6/conf
[hadoop@master conf]$ vim hive-site.xml
#在这个创建的配置文件中加入如下：
<configuration>
  <property>
    <name>javax.jdo.option.ConnectionURL</name>
    <value>jdbc:mysql://master:3306/hive?useSSL=false</value>
    <description>JDBC connect string for a JDBC metastore</description>
    <!-- master为你安装的MySQL服务器节点，请更改为自己的 -->
  </property>
  <property>
    <name>javax.jdo.option.ConnectionDriverName</name>
    <value>com.mysql.jdbc.Driver</value>
    <!-- SQL的驱动类名称，不用更改 -->
    <description>Driver class name for a JDBC metastore</description>
  </property>
  <property>
    <name>javax.jdo.option.ConnectionUserName</name>
    <value>hive</value>
    <!-- 连接MySQL的用户账户名 -->
    <description>username to use against metastore database</description>
  </property>
  <property>
    <name>javax.jdo.option.ConnectionPassword</name>
    <value>Hive@123456</value>
    <!-- 连接MySQL的用户的登录密码 -->
    <description>password to use against metastore database</description>
  </property>
  <property>
    <name>hive.metastore.warehouse.dir</name>
    <value>/home/hadoop/data/hive/warehouse</value>
    <description>hive default warehouse, if nessecory, change
it</description>
     <!--可选配置，该配置信息用来指定Hive数据仓库的数据存储在HDFS上的目录-->
  </property>

5、加入Mysql驱动

加入mysql驱动，mysql-connector-java-5.1.40-bin.jar，改jar包放置在hive的安装跟路径下的lib目录中。

[hadoop@master lib]$ pwd
/home/hadoop/apps/apache-hive-2.3.6/lib

6、复制Hadoop集群的配置文件

把Hadoop集群中的core-site.xml和hdfs-site.xml两个配置文件都放置在Hive安装目录下conf目录中。

[hadoop@master hadoop]$ cp core-site.xml /home/hadoop/apps/apache-hive-2.3.6/conf
[hadoop@master hadoop]$ cp hdfs-site.xml /home/hadoop/apps/apache-hive-2.3.6/conf

7、验证Hive安装

[hadoop@master hadoop]$ hive --service version
Hive 2.3.6
Git git://HW13934/Users/gates/tmp/hive-branch-2.3/hive -r 2c2fdd524e8783f6e1f3ef15281cc2d5ed08728f
Compiled by gates on Tue Aug 13 11:56:35 PDT 2019
From source with checksum c44b0b7eace3e81ba3cf64e7c4ea3b19

8、初始化元数据库

注意：当使用的hive是1.x之前的版本，不做初始化也是可以的，当Hive第一次启动的时候会自动进行初始化。如果使用的2.x版本的Hive，那么就必须手动初始化源数据库。使用命令：

[hadoop@master hadoop]$ schematool  -dbType mysql -initSchema
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/hadoop/apps/apache-hive-2.3.6/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/hadoop/apps/hadoop-2.7.7/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Metastore connection URL:	 jdbc:mysql://master:3306/hive?
createDatabaseIfNotExist=true&verifyServerCertificate=false&useSSL=false
Metastore Connection Driver :	 com.mysql.jdbc.Driver
Metastore connection User:	 hive
Starting metastore schema initialization to 2.3.0
Initialization script hive-schema-2.3.0.mysql.sql
Initialization script completed
schemaTool completed

注意：初始化操作只需要做一次。如果使用了一段时间，再执行了这个命令，那么就会又得到了一个权限的Hive。

9、启动Hive客户端、退出Hive

#方式一：
[hadoop@master hadoop]$ hive
which: no hbase in (/usr/local/jdk1.8.0_211/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/local/apache-maven-3.6.1/bin:/home/hadoop/.local/bin:/home/hadoop/bin:/home/hadoop/apps/hadoop-2.7.7/bin:/home/hadoop/apps/hadoop-2.7.7/sbin:/home/hadoop/apps/apache-hive-2.3.6/bin)
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/hadoop/apps/apache-hive-2.3.6/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/hadoop/apps/hadoop-2.7.7/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]

Logging initialized using configuration in jar:file:/home/hadoop/apps/apache-hive-2.3.6/lib/hive-common-2.3.6.jar!/hive-log4j2.properties Async: true
Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
hive> 

#方式二
[hadoop@master ~]$ hive --service cli
which: no hbase in (/usr/local/jdk1.8.0_211/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/local/apache-maven-3.6.1/bin:/home/hadoop/.local/bin:/home/hadoop/bin:/home/hadoop/apps/hadoop-2.7.7/bin:/home/hadoop/apps/hadoop-2.7.7/sbin:/home/hadoop/apps/apache-hive-2.3.6/bin)
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/hadoop/apps/apache-hive-2.3.6/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/hadoop/apps/hadoop-2.7.7/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]

Logging initialized using configuration in jar:file:/home/hadoop/apps/apache-hive-2.3.6/lib/hive-common-2.3.6.jar!/hive-log4j2.properties Async: true
Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
hive> 

#退出
hive> exit;
hive> quit;

10、注意事项

在使用Hive的时候要确保三件事：

（1）确保RDBMS元数据启动好了

（2）确保HDFS启动好了，能正常运行

（3）确保YARN集群启动好了，能正常运行

11、建表测试

hive> show databases;
OK
default
Time taken: 10.327 seconds, Fetched: 1 row(s)
hive> create database test;
OK
Time taken: 0.417 seconds
hive> show databases;
OK
default
test
Time taken: 0.029 seconds, Fetched: 2 row(s)
hive> use test;
OK
Time taken: 0.031 seconds
hive> create table test(x int);
OK
Time taken: 1.018 seconds
hive> show tables;
OK
test
Time taken: 0.037 seconds, Fetched: 1 row(s)
hive> 
#设置显示当前数据库
hive> set hive.cli.print.current.db=true;
hive (test)> 

12、Hive默认log文件保存位置

/tmp/<user.name>文件夹的hive.log文件中，全路径就是/tmp/当前用户名/hive.log。

[hadoop@master hadoop]$ pwd
/tmp/hadoop
#下面的hive.log文件

参考：

https://blog.csdn.net/wohiusdashi/article/details/89358071

https://www.cnblogs.com/yongjian/archive/2017/03/23/6607984.html