大数据平台实时数仓从0到1搭建之 - 05 hive 安装配置
概述
hive不需要集群配置,在server110上安装配置完,然后复制到server111,server112就行
环境
具体的环境信息可以参考02-架构设计篇
- Centos 7
- jdk 1.8
- hadoop-3.2.1
- hive-3.1.2
- mariadb-10.3
server110 192.168.1.110
server111 192.168.1.111
server112 192.168.1.112
集群规划
server110 | server111 | server112 | |
---|---|---|---|
hive | client | client | client |
mariadb | mysql |
mariadb10.3
安装
根据规划,安装到server111上,离线安装稍微有些浪费时间,就直接yum安装了
mariadb10.3 使用yum安装,需要配置repo,下面的repo是官网给的yum安装配置
[root@server111 ~]# cd /etc/yum.repos.d/
[root@server111 yum.repos.d]# vim mariadb.repo
[mariadb]
name = MariaDB
baseurl = http://yum.mariadb.org/10.3/centos7-amd64
gpgkey = https://yum.mariadb.org/RPM-GPG-KEY-MariaDB
gpgcheck = 1
[root@server111 yum.repos.d]# yum clean all
[root@server111 yum.repos.d]# yum -y update
[root@server111 yum.repos.d]# yum -y install MariaDB-client MariaDB-server
配置
[root@server111 yum.repos.d]# systemctl start mariadb #启动服务
[root@server111 yum.repos.d]# systemctl enable mariadb #设置开机启动
[root@server111 yum.repos.d]# mysql_secure_installation #快速初始化数据库
...
Enter current password for root (enter for none): #刚安装没密码,直接回车
OK, successfully used password, moving on...
...
Set root password? [Y/n] #是否设置root密码,直接回车
New password: #设置root新密码
Re-enter new password:
...
Remove anonymous users? [Y/n] #是否删除匿名登录,直接回车
...
Disallow root login remotely? [Y/n] #是否禁用root远程登录,直接回车
...
Remove test database and access to it? [Y/n] #是否删除test库
...
Reload privilege tables now? [Y/n] #重新加载配置
... Success!
Thanks for using MariaDB!
创建metastore库和hive用户
#登录
[root@server111 yum.repos.d]# mysql -uroot -proot
#切换数据库
MariaDB [(none)]> use mysql
#创建数据库metastore
MariaDB [mysql]> create database metastore;
#使用赋权语句,直接创建用户,将操作metastore库的所有权限赋权给密码为hive的hive用户,%表示可远程登录
MariaDB [mysql]> grant all on metastore.* to hive@'%' identified by 'hive';
MariaDB [mysql]> exit
Bye
hive
解压
回到server110
[root@server110 software]# tar -xzvf apache-hive-3.1.2-bin.tar.gz -C /opt/modules/
[root@server110 software]# cd ../modules/
[root@server110 modules]# mv apache-hive-3.1.2-bin/ apache-hive-3.1.2
hive-env.sh配置
[root@server110 modules]# cd apache-hive-3.1.2/conf/
[root@server110 conf]# mv hive-env.sh.template hive-env.sh
[root@server110 conf]# vim hive-env.sh
#shift+g 跳转到最后一行
#添加HADOOP_HOME和HIVE_CONF_DIR的环境变量
export HADOOP_HOME=/opt/modules/hadoop-3.2.1
export HIVE_CONF_DIR=/opt/modules/apache-hive-3.1.2/conf
hive-site.xml
hive-site.xml需要改的配置不多,所以直接创建这个文件,省的在template里搜了
就是配置jdbc那几个参数
[root@server110 conf]# touch hive-site.xml
[root@server110 conf]# vim hive-site.xml
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
<name>javax.jdo.option.ConnectionURL</name>
<value>jdbc:mysql://server111:3306/metastore?createDatabaseIfNotExist=true</value>
</property>
<property>
<name>javax.jdo.option.ConnectionPassword</name>
<value>hive</value>
</property>
<property>
<name>javax.jdo.option.ConnectionDriverName</name>
<value>com.mysql.jdbc.Driver</value>
</property>
<property>
<name>javax.jdo.option.ConnectionUserName</name>
<value>hive</value>
</property>
</configuration>
mysql jar包
然后将mysql驱动包上传到/opt/modules/apache-hive-3.1.2/lib下,
jar包我是随便从maven库里拿了一个
mysql-connector-java-5.1.13.jar
初始化元数据
初始化元数据报错
Exception in thread "main" java.lang.NoSuchMethodError: com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;Ljava/lang/Object;)V
[root@server110 apache-hive-3.1.2]# bin/schematool -dbType mysql -initSchema
`Exception in thread "main" java.lang.NoSuchMethodError: com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;Ljava/lang/Object;)V`
经查看是hadoop和hive 里guava包版本不一致导致,hadoop版本较高,hive较低,使用hadoop的替换hive对应的包
查看对应的jar
[root@server110 apache-hive-3.1.2]# ll lib/guava*
-rw-r--r--. 1 root root 2747878 10月 3 12:17 lib/guava-27.0-jre.jar
[root@server110 apache-hive-3.1.2]# ll /opt/modules/hadoop-3.2.1/share/hadoop/common/lib/guava*
-rw-r--r--. 1 test test 2747878 9月 10 2019 /opt/modules/hadoop-3.2.1/share/hadoop/common/lib/guava-27.0-jre.jar
替换高版本jar
[root@server110 apache-hive-3.1.2]# mv lib/guava-19.0.jar lib/guava-19.0.jar.bak
[root@server110 apache-hive-3.1.2]# cp /opt/modules/hadoop-3.2.1/share/hadoop/common/lib/guava-27.0-jre.jar lib/
初始化
[root@server110 apache-hive-3.1.2]# bin/schematool -dbType mysql -initSchema
..
Initialization script completed
schemaTool completed
配置完成
[root@server110 apache-hive-3.1.2]# bin/hive
which: no hbase in (/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/opt/modules/jdk1.8.0_181/bin:/opt/modules/hadoop-3.2.1/bin:/opt/modules/hadoop-3.2.1/sbin:/root/bin)
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/modules/apache-hive-3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/modules/hadoop-3.2.1/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Hive Session ID = 9d4bfb01-ffcf-4910-b026-6ab0e706b8c3
Logging initialized using configuration in jar:file:/opt/modules/apache-hive-3.1.2/lib/hive-common-3.1.2.jar!/hive-log4j2.properties Async: true
Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
hive>
测试hive
创建test库
hive> create database test;
OK
Time taken: 0.129 seconds
hive> show databases;
OK
default
test
Time taken: 0.053 seconds, Fetched: 2 row(s)
同步到server111,server112
[root@server110 modules]# scp -r apache-hive-3.1.2/ server111:/opt/modules/
[root@server110 modules]# scp -r apache-hive-3.1.2/ server112:/opt/modules/
server111测试
[root@server111 ~]# cd /opt/modules/apache-hive-3.1.2/
[root@server111 apache-hive-3.1.2]# bin/hive
hive> show databases;
OK
default
test
Time taken: 1.171 seconds, Fetched: 2 row(s)
server112测试
[root@server112 ~]# cd /opt/modules/apache-hive-3.1.2/
[root@server112 apache-hive-3.1.2]# bin/hive
hive> show databases;
OK
default
test
Time taken: 0.826 seconds, Fetched: 2 row(s)