Hive的安装及配置

一、Hive安装地址

1、Hive官网地址
http://hive.apache.org/

2、文档查看地址
https://cwiki.apache.org/confluence/display/Hive/GettingStarted

3、下载地址
http://archive.apache.org/dist/hive/

二、Hive安装部署

2.1 Hive安装及配置

1、把 apache-hive-1.2.1-bin.tar.gz 上传到 linux 的 /opt/software 目录下

[test@hadoop151 opt]$ cd software/
[test@hadoop151 software]$ ll
总用量 464860
-rw-rw-r--  1 test test  92834839 2月  14 09:48 apache-hive-1.2.1-bin.tar.gz
-rw-rw-r--. 1 test test 197657687 1月  25 10:41 hadoop-2.7.2.tar.gz
-rw-rw-r--. 1 test test 185515842 1月  25 09:33 jdk-8u144-linux-x64.tar.gz

2、解压 apache-hive-1.2.1-bin.tar.gz 到 /opt/module/ 目录下面

[test@hadoop151 software]$ tar -zxvf apache-hive-1.2.1-bin.tar.gz  -C /opt/module/

3、修改 apache-hive-1.2.1-bin.tar.gz 的名称为 hive

[test@hadoop151 module]$ mv apache-hive-1.2.1-bin/ hive
[test@hadoop151 module]$ ll
总用量 16
drwxr-xr-x. 15 test test 4096 2月   3 21:53 hadoop-2.7.2
drwxrwxr-x   8 test test 4096 2月  14 09:49 hive
drwxrwxr-x   2 test test 4096 2月   4 19:16 input
drwxr-xr-x.  8 test test 4096 7月  22 2017 jdk1.8.0_144

4、修改/opt/module/hive/conf 目录下的 hive-env.sh.template 名称为 hive-env.sh

[test@hadoop151 conf]$ mv hive-env.sh.template hive-env.sh

5、配置 hive-env.sh 文件
(1) 配置 HADOOP_HOME 路径

export HADOOP_HOME=/opt/module/hadoop-2.7.2

(2) 配置 HIVE_CONF_DIR 路径

export HIVE_CONF_DIR=/opt/module/hive/conf 

2.2 Hadoop集群配置

1、必须启动 hdfs 和 yarn

[test@hadoop151 conf]$ start-dfs.sh
Starting namenodes on [hadoop151]
hadoop151: starting namenode, logging to /opt/module/hadoop-2.7.2/logs/hadoop-test-namenode-hadoop151.out
hadoop151: starting datanode, logging to /opt/module/hadoop-2.7.2/logs/hadoop-test-datanode-hadoop151.out
hadoop152: starting datanode, logging to /opt/module/hadoop-2.7.2/logs/hadoop-test-datanode-hadoop152.out
hadoop153: starting datanode, logging to /opt/module/hadoop-2.7.2/logs/hadoop-test-datanode-hadoop153.out
Starting secondary namenodes [hadoop153]
hadoop153: starting secondarynamenode, logging to /opt/module/hadoop-2.7.2/logs/hadoop-test-secondarynamenode-hadoop153.out

[test@hadoop152 ~]$ start-yarn.sh
starting yarn daemons
starting resourcemanager, logging to /opt/module/hadoop-2.7.2/logs/yarn-test-resourcemanager-hadoop152.out
hadoop153: starting nodemanager, logging to /opt/module/hadoop-2.7.2/logs/yarn-test-nodemanager-hadoop153.out
hadoop151: starting nodemanager, logging to /opt/module/hadoop-2.7.2/logs/yarn-test-nodemanager-hadoop151.out
hadoop152: starting nodemanager, logging to /opt/module/hadoop-2.7.2/logs/yarn-test-nodemanager-hadoop152.out

2、在 HDFS 上创建 /tmp 和 /user/hive/warehouse 两个目录并修改他们的同组权限可写(可不操作，系统会自动创建)

[test@hadoop153 ~]$ hadoop fs -mkdir /tmp
[test@hadoop153 ~]$ hadoop fs -mkdir -p /user/hive/warehouse

[test@hadoop153 ~]$ hadoop fs -chmod g+w /tmp
[test@hadoop153 ~]$ hadoop fs -chmod g+w /user/hive/warehouse

2.3 Hive基本操作

1、启动hive

[test@hadoop151 bin]$ hive
Logging initialized using configuration in jar:file:/opt/module/hive/lib/hive-common-1.2.1.jar!/hive-log4j.properties
hive> 

2、查看数据库

hive> show databases;
OK
default
Time taken: 1.355 seconds, Fetched: 1 row(s)

3、打开默认的数据库

hive> use default;
OK
Time taken: 0.075 seconds

4、显示 default 数据库中的表

hive> show tables;
OK
Time taken: 0.063 seconds

5、创建一张表

hive> create table student(id int, name string);
OK
Time taken: 0.584 seconds

6、显示数据库中有几张表

hive> show tables;
OK
student
Time taken: 0.022 seconds, Fetched: 1 row(s)

7、查看表的结构

hive> desc student;
OK
id                  	int                 	                    
name                	string              	                    
Time taken: 0.206 seconds, Fetched: 2 row(s)

8、向表中插入数据

hive> insert into student values(1000,"ss");
Query ID = test_20200214104152_6f97830a-f3f8-4c26-9744-c77441cefd05
Total jobs = 3
Launching Job 1 out of 3
Number of reduce tasks is set to 0 since there's no reduce operator
Starting Job = job_1581645695424_0001, Tracking URL = http://hadoop152:8088/proxy/application_1581645695424_0001/
Kill Command = /opt/module/hadoop-2.7.2/bin/hadoop job  -kill job_1581645695424_0001
Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 0
2020-02-14 10:42:18,853 Stage-1 map = 0%,  reduce = 0%
2020-02-14 10:42:38,866 Stage-1 map = 100%,  reduce = 0%, Cumulative CPU 4.2 sec
MapReduce Total cumulative CPU time: 4 seconds 200 msec
Ended Job = job_1581645695424_0001
Stage-4 is selected by condition resolver.
Stage-3 is filtered out by condition resolver.
Stage-5 is filtered out by condition resolver.
Moving data to: hdfs://hadoop151:9000/user/hive/warehouse/student/.hive-staging_hive_2020-02-14_10-41-52_166_5255955701507975136-1/-ext-10000
Loading data to table default.student
Table default.student stats: [numFiles=1, numRows=1, totalSize=8, rawDataSize=7]
MapReduce Jobs Launched: 
Stage-Stage-1: Map: 1   Cumulative CPU: 4.2 sec   HDFS Read: 3557 HDFS Write: 79 SUCCESS
Total MapReduce CPU Time Spent: 4 seconds 200 msec
OK
Time taken: 49.347 seconds

9、查询表中数据

hive> select * from student;
OK
1000	ss
Time taken: 0.098 seconds, Fetched: 1 row(s)

10、退出hive

hive> quit;

三、使用MySQL存储Metastore

3.1 为何使用MySQL储存Metastore？

如果我们已经打开了一个 hive 窗口，那么当我们再打开一个客户端启动 hive，会产生 java.sql.SQLException 异常

Exception in thread "main" java.lang.RuntimeException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient

原因是：Metastore 默认存储在自带的 derby 数据库中，推荐使用 MySQL 存储 Metastore。

3.2 MySQL安装

1、查看 mysql 是否安装，如果安装了，卸载 mysql

(1) 查看

[test@hadoop151 ~]$ rpm -qa | grep mysql
mysql-libs-5.1.73-7.e