大数据平台实时数仓从0到1搭建之 - 05 hive 安装配置

概述

hive不需要集群配置，在server110上安装配置完，然后复制到server111,server112就行

环境
具体的环境信息可以参考02-架构设计篇

• Centos 7
• jdk 1.8
• hadoop-3.2.1
• hive-3.1.2
• mariadb-10.3

server110 192.168.1.110
server111 192.168.1.111
server112 192.168.1.112

集群规划

	server110	server111	server112
hive	client	client	client
mariadb		mysql

mariadb10.3

安装

根据规划，安装到server111上，离线安装稍微有些浪费时间，就直接yum安装了
mariadb10.3 使用yum安装，需要配置repo，下面的repo是官网给的yum安装配置

[root@server111 ~]# cd /etc/yum.repos.d/
[root@server111 yum.repos.d]# vim mariadb.repo
[mariadb]
name = MariaDB
baseurl = http://yum.mariadb.org/10.3/centos7-amd64
gpgkey = https://yum.mariadb.org/RPM-GPG-KEY-MariaDB
gpgcheck = 1
[root@server111 yum.repos.d]# yum clean all
[root@server111 yum.repos.d]# yum -y update
[root@server111 yum.repos.d]# yum -y install MariaDB-client MariaDB-server

配置

[root@server111 yum.repos.d]# systemctl start mariadb  #启动服务
[root@server111 yum.repos.d]# systemctl enable mariadb	#设置开机启动
[root@server111 yum.repos.d]# mysql_secure_installation	#快速初始化数据库
...
Enter current password for root (enter for none): #刚安装没密码，直接回车
OK, successfully used password, moving on...
...
Set root password? [Y/n] #是否设置root密码，直接回车
New password: #设置root新密码
Re-enter new password: 
...
Remove anonymous users? [Y/n] #是否删除匿名登录，直接回车
 ... 
Disallow root login remotely? [Y/n] #是否禁用root远程登录，直接回车
 ...
Remove test database and access to it? [Y/n] #是否删除test库
...
Reload privilege tables now? [Y/n] #重新加载配置
 ... Success!
 
Thanks for using MariaDB!

创建metastore库和hive用户

#登录
[root@server111 yum.repos.d]# mysql -uroot -proot
#切换数据库	
MariaDB [(none)]> use mysql		
#创建数据库metastore
MariaDB [mysql]> create database metastore;		
#使用赋权语句，直接创建用户，将操作metastore库的所有权限赋权给密码为hive的hive用户，%表示可远程登录
MariaDB [mysql]> grant all on metastore.* to  hive@'%' identified by 'hive';	
MariaDB [mysql]> exit
Bye

hive

解压

回到server110

[root@server110 software]# tar -xzvf apache-hive-3.1.2-bin.tar.gz -C /opt/modules/
[root@server110 software]# cd ../modules/
[root@server110 modules]# mv apache-hive-3.1.2-bin/ apache-hive-3.1.2

hive-env.sh配置

[root@server110 modules]# cd apache-hive-3.1.2/conf/
[root@server110 conf]# mv hive-env.sh.template hive-env.sh
[root@server110 conf]# vim hive-env.sh
#shift+g 跳转到最后一行
#添加HADOOP_HOME和HIVE_CONF_DIR的环境变量
export HADOOP_HOME=/opt/modules/hadoop-3.2.1
export HIVE_CONF_DIR=/opt/modules/apache-hive-3.1.2/conf

hive-site.xml

hive-site.xml需要改的配置不多，所以直接创建这个文件，省的在template里搜了
就是配置jdbc那几个参数

[root@server110 conf]# touch hive-site.xml
[root@server110 conf]# vim hive-site.xml
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
  <property>
    <name>javax.jdo.option.ConnectionURL</name>
    <value>jdbc:mysql://server111:3306/metastore?createDatabaseIfNotExist=true</value>
  </property>
  <property>
    <name>javax.jdo.option.ConnectionPassword</name>
    <value>hive</value>
  </property>
  <property>
    <name>javax.jdo.option.ConnectionDriverName</name>
    <value>com.mysql.jdbc.Driver</value>
  </property>
  <property>
    <name>javax.jdo.option.ConnectionUserName</name>
    <value>hive</value>
  </property>
</configuration>

mysql jar包

然后将mysql驱动包上传到/opt/modules/apache-hive-3.1.2/lib下，
jar包我是随便从maven库里拿了一个
mysql-connector-java-5.1.13.jar

初始化元数据

初始化元数据报错
Exception in thread "main" java.lang.NoSuchMethodError: com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;Ljava/lang/Object;)V

[root@server110 apache-hive-3.1.2]# bin/schematool -dbType mysql -initSchema
`Exception in thread "main" java.lang.NoSuchMethodError: com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;Ljava/lang/Object;)V`

经查看是hadoop和hive 里guava包版本不一致导致，hadoop版本较高，hive较低，使用hadoop的替换hive对应的包

查看对应的jar

[root@server110 apache-hive-3.1.2]# ll lib/guava*
-rw-r--r--. 1 root root 2747878 10月  3 12:17 lib/guava-27.0-jre.jar
[root@server110 apache-hive-3.1.2]# ll /opt/modules/hadoop-3.2.1/share/hadoop/common/lib/guava*
-rw-r--r--. 1 test test 2747878 9月  10 2019 /opt/modules/hadoop-3.2.1/share/hadoop/common/lib/guava-27.0-jre.jar

替换高版本jar

[root@server110 apache-hive-3.1.2]# mv lib/guava-19.0.jar lib/guava-19.0.jar.bak
[root@server110 apache-hive-3.1.2]# cp /opt/modules/hadoop-3.2.1/share/hadoop/common/lib/guava-27.0-jre.jar lib/

初始化

[root@server110 apache-hive-3.1.2]# bin/schematool -dbType mysql -initSchema
..
Initialization script completed
schemaTool completed

配置完成

[root@server110 apache-hive-3.1.2]# bin/hive
which: no hbase in (/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/opt/modules/jdk1.8.0_181/bin:/opt/modules/hadoop-3.2.1/bin:/opt/modules/hadoop-3.2.1/sbin:/root/bin)
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/modules/apache-hive-3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/modules/hadoop-3.2.1/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Hive Session ID = 9d4bfb01-ffcf-4910-b026-6ab0e706b8c3

Logging initialized using configuration in jar:file:/opt/modules/apache-hive-3.1.2/lib/hive-common-3.1.2.jar!/hive-log4j2.properties Async: true
Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
hive>

测试hive

创建test库

hive> create database test;
OK
Time taken: 0.129 seconds
hive> show databases;
OK
default
test
Time taken: 0.053 seconds, Fetched: 2 row(s)

同步到server111,server112

[root@server110 modules]# scp -r apache-hive-3.1.2/ server111:/opt/modules/
[root@server110 modules]# scp -r apache-hive-3.1.2/ server112:/opt/modules/

server111测试

[root@server111 ~]# cd /opt/modules/apache-hive-3.1.2/
[root@server111 apache-hive-3.1.2]# bin/hive
hive> show databases;
OK
default
test
Time taken: 1.171 seconds, Fetched: 2 row(s)

server112测试

[root@server112 ~]# cd /opt/modules/apache-hive-3.1.2/
[root@server112 apache-hive-3.1.2]# bin/hive
hive> show databases;
OK
default
test
Time taken: 0.826 seconds, Fetched: 2 row(s)

完成