Hadoop分布式 Hive 搭建

说明：
Hive 搭建需要借助 MySQL 存储元数据，因此要安装 MySQL 具体参考： MySQL 安装
Hive 是建立在 Hadoop 上的，因此需要先搭建 Hadoop 具体参考：Hadoop 搭建

注意：Hive只需在Hadoop集群的NameNode节点上安装即可，无需在DataNode节点上安装

下载并解压hive安装包到 /opt 目录下 改变所属用户为当前用户

下载地址

[qgn@qq1 opt]$ sudo mv apache-hive-2.3.5-bin hive-2.3.5
[qgn@qq1 opt]$ ll
总用量 0
drwxr-xr-x. 11 qgn  qgn  173 5月  24 11:14 hadoop-2.9.2
drwxr-xr-x. 10 root root 184 5月  24 15:26 hive-2.3.5
[qgn@qq1 opt]$ sudo chown -R qgn:qgn hive-2.3.5
[qgn@qq1 opt]$ ll
总用量 0
drwxr-xr-x. 11 qgn qgn 173 5月  24 11:14 hadoop-2.9.2
drwxr-xr-x. 10 qgn qgn 184 5月  24 15:26 hive-2.3.5
[qgn@qq1 opt]$ ls
 

配置环境变量

将下面三行加入到 /etc/profile 文件末尾 source /etc/profile 使配置文件生效

export HIVE_HOME=/opt/hive-2.3.5
export HIVE_CONF_DIR=$HIVE_HOME/conf
export PATH=$PATH:$HIVE_HOME/bin

在HDFS中创建Hive所需目录 并授权，用于存储文件

[qgn@qq1 hive-2.3.5]$ hdfs dfs -mkdir -p /user/hive/warehouse
[qgn@qq1 hive-2.3.5]$ hdfs dfs -chmod -R 777 /user/hive/warehouse
[qgn@qq1 hive-2.3.5]$ hdfs dfs -mkdir -p /tmp/hive
[qgn@qq1 hive-2.3.5]$ hdfs dfs -chmod -R 777 /tmp/hive
[qgn@qq1 hive-2.3.5]$ hdfs dfs -ls /
Found 2 items
drwxr-xr-x   - qgn supergroup          0 2019-05-24 17:36 /tmp
drwxr-xr-x   - qgn supergroup          0 2019-05-24 17:35 /user
[qgn@qq1 hive-2.3.5]$ 

MySQL 的驱动包 下载安装

下载地址
将 MySQL 的驱动包解压将 mysql-connector-java-5.1.47.jar 拷贝到 /opt/hive-2.3.5/lib 目录下

[qgn@qq1 ~]$ ls
mysql-connector-java-5.1.47.tar.gz
[qgn@qq1 ~]$ tar zxf mysql-connector-java-5.1.47.tar.gz
[qgn@qq1 ~]$ ls
mysql-connector-java-5.1.47  mysql-connector-java-5.1.47.tar.gz
[qgn@qq1 ~]$ cd mysql-connector-java-5.1.47/
[qgn@qq1 mysql-connector-java-5.1.47]$ ls
build.xml  CHANGES  COPYING  mysql-connector-java-5.1.47-bin.jar  mysql-connector-java-5.1.47.jar  README  README.txt  src
[qgn@qq1 mysql-connector-java-5.1.47]$ ls
build.xml  CHANGES  COPYING  mysql-connector-java-5.1.47-bin.jar  mysql-connector-java-5.1.47.jar  README  README.txt  src
[qgn@qq1 mysql-connector-java-5.1.47]$ cp mysql-connector-java-5.1.47.jar $HIVE_HOME/lib

配置 文件在 hive-2.3.5/conf/ 下

将 hive-env.sh.template 复制一份命名为 hive-env.sh
将 hive-default.xml.template 复制一份命名为 hive-site.xml

[qgn@qq1 opt]$ cd hive-2.3.5/conf/
[qgn@qq1 conf]$ ls
beeline-log4j2.properties.template    ivysettings.xml
hive-default.xml.template             llap-cli-log4j2.properties.template
hive-env.sh.template                  llap-daemon-log4j2.properties.template
hive-exec-log4j2.properties.template  parquet-logging.properties
hive-log4j2.properties.template
[qgn@qq1 conf]$ cp hive-default.xml.template hive-site.xml
[qgn@qq1 conf]$ cp hive-env.sh.template hive-env.sh

建议修改配置文件用 Notepad++ 可以连接 liunx 更改配置文件
具体参考：https://blog.csdn.net/louniuous/article/details/80220559

配置 hive-env.sh

# Set HADOOP_HOME to point to a specific hadoop install directory
HADOOP_HOME=/opt/hadoop-2.9.2
 
# Hive Configuration Directory can be controlled by:
export HIVE_CONF_DIR=/opt/hive-2.3.5/conf
 
# Folder containing extra libraries required for hive compilation/execution can be controlled by:
export HIVE_AUX_JARS_PATH=/opt/hive-2.3.5/lib

配置hive-site.xml 文件

可以看到上面新建的文件使用的是默认配置，此处不许修改

<property>
    <name>hive.metastore.warehouse.dir</name>
    <value>/user/hive/warehouse</value>
    <description>location of default database for the warehouse</description>
</property>
<property>
    <name>hive.exec.scratchdir</name>
    <value>/tmp/hive</value>
    <description>HDFS root scratch dir for Hive jobs which gets created with write all (733) permission. </description>
</property>

搜索所有 ${system:java.io.tmpdir} 替换为本地hive临时文件，我的事 /opt/hive-2.3.5/tmp
搜索所有 ${system:user.name} 替换为hive 所属用户的用户名 我的是 qgn

配置 MySQL

属性名称	描述
javax.jdo.option.ConnectionDriverName	数据库的驱动类名称
javax.jdo.option.ConnectionURL	数据库的JDBC连接地址
javax.jdo.option.ConnectionUserName	连接数据库所使用的用户名
javax.jdo.option.ConnectionPassword	连接数据库所使用的密码

搜索配置名 javax.jdo.option.ConnectionDriverName ，将对应的 value 修改为MySQL的驱动类名称 com.mysql.jdbc.Driver
搜索配置名 javax.jdo.option.connectionURL ，将对应的 value 修改为MySQL的地址
搜索配置名 javax.jdo.option.ConnectionUserName，将对应的 value 修改为MySQL数据库登录名 root
搜索配置名 javax.jdo.option.ConnectionPassword，将对应的 value 修改为MySQL数据库的登录密码 我的是 mysql

<property>
	 <name>javax.jdo.option.ConnectionDriverName</name>
	 <value>com.mysql.jdbc.Driver</value>
	 <description>Driver class name for a JDBC metastore</description>
</property>
<property>
	<name>javax.jdo.option.ConnectionURL</name>
	<value>jdbc:mysql://qq1:3306/hive?createDatabaseIfNotExist=true&amp;useSSL=false</value>
	<description>
	  JDBC connect string for a JDBC metastore.
	  To use SSL to encrypt/authenticate the connection, provide database-specific SSL flag in the connection URL.
	  For example, jdbc:postgresql://myhost/db?ssl=true for postgres database.
	</description>
</property>
<property>
	<name>javax.jdo.option.ConnectionUserName</name>
	<value>root</value>
	<description>Username to use against metastore database</description>
</property>
<property>
	<name>javax.jdo.option.ConnectionPassword</name>
	<value>mysql</value>
	<description>password to use against metastore database</description>
</property>

其他配置

在hive界面显示当前数据库名字
显示列名

<property>
   <name>hive.cli.print.current.db</name>
   <value>true</value>
   <description>Whether to include the current database in the Hive prompt.</description>
</property>
<property>
   <name>hive.cli.print.header</name>
   <value>true</value>
   <description>Whether to print the names of the columns in query output.</description>
</property>

初始化数据库 并启动 hive

在 bin 目录下执行 schematool -initSchema -dbType mysql 初始化
执行 hive 启动

[qgn@qq1 bin]$ schematool -initSchema -dbType mysql
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/hive-2.3.5/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/hadoop-2.7.3/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Metastore connection URL:        jdbc:mysql://qq1:3306/hive?createDatabaseIfNotExist=true
Metastore Connection Driver :    com.mysql.jdbc.Driver
Metastore connection User:       root
Starting metastore schema initialization to 2.3.0
Initialization script hive-schema-2.3.0.mysql.sql
Initialization script completed
schemaTool completed
[qgn@qq1 bin]$ hive
which: no hbase in (/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/hadoop-2.7.3/bin:/opt/hadoop-2.7.3/sbin:/usr/local/lib64/jdk1.8.0_211/bin:/usr/local/mysql/bin:/home/qgn/.local/bin:/home/qgn/bin:HIVE_HOME/bin:/opt/hive-2.3.5/bin)
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/hive-2.3.5/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/hadoop-2.7.3/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]

Logging initialized using configuration in jar:file:/opt/hive-2.3.5/lib/hive-common-2.3.5.jar!/hive-log4j2.properties Async: true
Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
hive> create database user_test;
OK
Time taken: 4.491 seconds
hive> use  user_test;
OK
Time taken: 0.032 seconds
hive> quit;
[qgn@qq1 bin]$