linux 下安装hive-2.3.7包含配置mysql以及hadoop

---

前提：MySQL以及hadoop已经安装好，并且都已经正常启动

Hadoop启动查看jps这样即正常

[root@localhost conf]# jps

1952 SecondaryNameNode

1527 NameNode

2104 ResourceManager

2393 NodeManager

18505 Jps

17039 RunJar

---

1.安装包下载

下载地址http://mirrors.hust.edu.cn/apache/hive/hive-2.3.7/

 

2.解压安装包

[root@localhost hive]# tar -axf apache-hive-2.3.7-bin

3.修改配置文件hive-site.xml 

配置文件路径：

[root@localhost conf]# pwd

/root/hive/apache-hive-2.3.7-bin/conf

新建文件夹并且添加内容

[root@localhost conf]# touch hive-site.xml

[root@localhost conf]# vim hive-site.xml

<configuration>

       <property>

                <name>javax.jdo.option.ConnectionURL</name>

                <value>jdbc:mysql://localhost:3306/hive?characterEncoding=UTF-8</value>

                <description>JDBC connect string for a JDBC metastore</description>

        </property>

        <property>

                <name>javax.jdo.option.ConnectionDriverName</name>

                <value>com.mysql.jdbc.Driver</value>

                <description>Driver class name for a JDBC metastore</description>

        </property>

        <property>

                <name>javax.jdo.option.ConnectionUserName</name>

                <value>root</value>

                <description>username to use against metastore database</description>

        </property>

        <property>

                <name>javax.jdo.option.ConnectionPassword</name>

                <value>root</value>

        <description>password to use against metastore database</description>

        </property>

</configuration>

4.修改文件hive-env.sh

文件路径：

[root@localhost conf]# pwd

/root/hive/apache-hive-2.3.7-bin/conf

文件尾部加入：

export HADOOP_HOME=/opt/app/hadoop-2.7.7

5.将mysql驱动jar包放到hive的lib目录下

[root@localhost lib]# pwd

/root/hive/apache-hive-2.3.7-bin/lib

注：此jar包需要自行下载，并非mysql 自带。

下载地址：

https://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java-5.1.46.zip

 

 

6.修改环境变量

查看hive_home:

[root@localhost conf]# echo $HIVE_HOME

/root/hive/apache-hive-2.3.7-bin

查看hadoop_home:

[root@localhost conf]# echo $HADOOP_HOME

/opt/app/hadoop-2.7.7

文件中加入如下内容：

export HADOOP_HOME=/opt/app/hadoop-2.7.7

export PATH=$HADOOP_HOME/bin:$PATH

export HIVE_HOME=/root/hive/apache-hive-2.3.7-bin

export PATH=$PATH:$HIVE_HOME/bin

 

修改完配置文件source 一下

[root@localhost lib]# source ~/.bashrc

 

7.验证hive 安装

[root@localhost conf]# hive --help

Usage ./hive <parameters> --service serviceName <service parameters>

Service List: beeline cleardanglingscratchdir cli hbaseimport hbaseschematool help hiveburninclient hiveserver2 hplsql jar lineage llapdump llap llapstatus metastore metatool orcfiledump rcfilecat schemaTool version

Parameters parsed:

  --auxpath : Auxiliary jars

  --config : Hive configuration directory

  --service : Starts specific service/component. cli is default

Parameters used:

  HADOOP_HOME or HADOOP_PREFIX : Hadoop install directory

  HIVE_OPT : Hive options

For help on a particular service:

  ./hive --service serviceName --help

Debug help:  ./hive --debug --help

 

8.mysql数据库配置

mysql> use mysql;

Reading table information for completion of table and column names

You can turn off this feature to get a quicker startup with -A

 

Database changed

mysql> create database hive charset=utf8;

Query OK, 1 row affected (0.02 sec)

 

mysql> GRANT ALL PRIVILEGES ON *.* TO 'root'@'%' IDENTIFIED BY 'PASSWORD' WITH GRANT OPTION ;

Query OK, 0 rows affected (0.01 sec)

 

mysql> quit

 

9.初始化元数据库

注意：当使用的 hive 是 2.x 之前的版本，不做初始化也是 OK 的，当 hive 第一次启动的 时候会自动进行初始化，只不过会不会生成足够多的元数据库中的表。在使用过程中会 慢慢生成。但最后进行初始化。如果使用的 2.x 版本的 Hive，那么就必须手动初始化元 数据库。使用命令：

schematool -dbType mysql -initSchema

[root@localhost conf]# schematool -dbType mysql -initSchema

SLF4J: Class path contains multiple SLF4J bindings.

SLF4J: Found binding in [jar:file:/root/hive/apache-hive-2.3.7-bin/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: Found binding in [jar:file:/opt/app/hadoop-2.7.7/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.

SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]

Metastore connection URL:  jdbc:mysql://localhost:3306/hive?characterEncoding=UTF-8

Metastore Connection Driver :  com.mysql.jdbc.Driver

Metastore connection User:  root

Starting metastore schema initialization to 2.3.0

Initialization script hive-schema-2.3.0.mysql.sql

Initialization script completed

schemaTool completed

 

 

10.检查数据库

mysql> show databases;

+--------------------+

| Database           |

+--------------------+

| information_schema |

| hive               |

| mysql              |

| performance_schema |

| test               |

+--------------------+

5 rows in set (0.01 sec)

 

mysql> use hive

Reading table information for completion of table and column names

You can turn off this feature to get a quicker startup with -A

 

Database changed

mysql> use hive;

Database changed

mysql> show tables;

+---------------------------+

| Tables_in_hive            |

+---------------------------+

| AUX_TABLE                 |

| BUCKETING_COLS            |

| CDS                       |

| COLUMNS_V2                |

| COMPACTION_QUEUE          |

| COMPLETED_COMPACTIONS     |

| COMPLETED_TXN_COMPONENTS  |

| DATABASE_PARAMS           |

| DBS                       |

| DB_PRIVS                  |

| DELEGATION_TOKENS         |

| FUNCS                     |

| FUNC_RU                   |

| GLOBAL_PRIVS              |

| HIVE_LOCKS                |

| IDXS                      |

| INDEX_PARAMS              |

| KEY_CONSTRAINTS           |

| MASTER_KEYS               |

| NEXT_COMPACTION_QUEUE_ID  |

| NEXT_LOCK_ID              |

| NEXT_TXN_ID               |

| NOTIFICATION_LOG          |

| NOTIFICATION_SEQUENCE     |

| NUCLEUS_TABLES            |

| PARTITIONS                |

| PARTITION_EVENTS          |

| PARTITION_KEYS            |

| PARTITION_KEY_VALS        |

| PARTITION_PARAMS          |

| PART_COL_PRIVS            |

| PART_COL_STATS            |

| PART_PRIVS                |

| ROLES                     |

| ROLE_MAP                  |

| SDS                       |

| SD_PARAMS                 |

| SEQUENCE_TABLE            |

| SERDES                    |

| SERDE_PARAMS              |

| SKEWED_COL_NAMES          |

| SKEWED_COL_VALUE_LOC_MAP  |

| SKEWED_STRING_LIST        |

| SKEWED_STRING_LIST_VALUES |

| SKEWED_VALUES             |

| SORT_COLS                 |

| TABLE_PARAMS              |

| TAB_COL_STATS             |

| TBLS                      |

| TBL_COL_PRIVS             |

| TBL_PRIVS                 |

| TXNS                      |

| TXN_COMPONENTS            |

| TYPES                     |

| TYPE_FIELDS               |

| VERSION                   |

| WRITE_SET                 |

+---------------------------+

57 rows in set (0.00 sec)

 

 

11.在hadoop下创建hive所用文件夹

前提,配置了本机HADOOP_HOME的环境变量，可以这么使用命令.
创建文件夹 并赋予权限.

$HADOOP_HOME/bin/hadoop fs -mkdir  -p     /tmp

$HADOOP_HOME/bin/hadoop fs -mkdir  -p     /user/hive/warehouse

$HADOOP_HOME/bin/hadoop fs -chmod g+w   /tmp

$HADOOP_HOME/bin/hadoop fs -chmod g+w   /user/hive/warehouse

 

发现创建时候报错：大致意思hadoop是处于安全模式,关闭安全模式

[root@localhost bin]# $HADOOP_HOME/bin/hadoop fs -mkdir  -p     /user/hive/warehouse

mkdir: Cannot create directory /user/hive/warehouse. Name node is in safe mode.

 

[root@localhost bin]# hadoop  dfsadmin -safemode leave

DEPRECATED: Use of this script to execute hdfs command is deprecated.

Instead use the hdfs command for it.

Safe mode is OFF

 

继续创建即可

 

12.进入hive

[root@localhost bin]# hive

which: no hbase in (/opt/app/hadoop-2.7.7/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/usr/java/jdk1.8.0_131/bin:/usr/java/jdk1.8.0_131/jre/bin:/root/hive/apache-hive-2.3.7-bin/bin:/root/bin:/usr/local/python3/bin:/usr/java/jdk1.8.0_131/bin:/usr/java/jdk1.8.0_131/jre/bin:/usr/java/jdk1.8.0_131/bin:/usr/java/jdk1.8.0_131/jre/bin:/usr/java/jdk1.8.0_131/bin:/usr/java/jdk1.8.0_131/jre/bin:/usr/java/jdk1.8.0_131/bin:/usr/java/jdk1.8.0_131/jre/bin:/usr/java/jdk1.8.0_131/bin:/usr/java/jdk1.8.0_131/jre/bin:/root/hive/apache-hive-2.3.7-bin/bin:/usr/java/jdk1.8.0_131/bin:/usr/java/jdk1.8.0_131/jre/bin:/usr/java/jdk1.8.0_131/bin:/usr/java/jdk1.8.0_131/jre/bin:/root/hive/apache-hive-2.3.7-bin/bin)

SLF4J: Class path contains multiple SLF4J bindings.

SLF4J: Found binding in [jar:file:/root/hive/apache-hive-2.3.7-bin/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: Found binding in [jar:file:/opt/app/hadoop-2.7.7/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.

SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]

 

Logging initialized using configuration in jar:file:/root/hive/apache-hive-2.3.7-bin/lib/hive-common-2.3.7.jar!/hive-log4j2.properties Async: true

Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.

hive>

 

 

13.创建一个数据库myhive

hive> create database myhive;

OK

Time taken: 7.153 seconds

 

14.使用新的数据库myhive

 

hive> use myhive;

OK

Time taken: 0.065 seconds

 

15.查看正在使用的数据库

 

hive> select current_database();

OK

myhive

Time taken: 0.631 seconds, Fetched: 1 row(s)