hive安装个人使用文档之hive搭建

前提

①hadoop集群，
②mysql
③zookeeper

单机版hive（derby作为元数据库）

解压重命名

[root@hadoop100 opt]# ll
total 40
drwxr-xr-x 12 root root   183 Sep 15 15:28 hadoop
drwxr-xr-x  8 root root   159 Sep 24 14:09 hive
drwxr-xr-x  8 root root   255 Sep 15 15:23 jdk
drwxr-xr-x  2 root root   146 Sep 24 15:08 myself
drwxr-xr-x  2 root root   146 Sep 24 14:44 soft
drwxr-xr-x 11 1001 1001  4096 Sep 24 14:48 zookeeper
-rw-r--r--  1 root root 32926 Sep 24 15:11 zookeeper.out
[root@hadoop100 opt]# 

配置环境变量

#  配置hive环境变量
export HIVE_HOME=/opt/hive
export PATH=$PATH:$HIVE_HOME/bin

source生效

配置配置文件

hive-env

[root@hadoop100 opt]# cd hive/conf/
[root@hadoop100 conf]# cp hive-env.sh.template hive-env.sh
[root@hadoop100 conf]# vim hive-env.sh

export JAVA_HOME=/opt/jdk
export HADOOP_HOME=/opt/hadoop
export HIVE_HOME=/opt/hive
export HIVE_CONF_DIR=/opt/hive/conf

hive-site

[root@hadoop100 conf]# cp hive-default.xml.template hive-site.xml
[root@hadoop100 conf]# vim hive-site.xml 

<configuration>  
<property>  
  <name>javax.jdo.option.ConnectionURL</name>  
  <value>jdbc:derby:;databaseName=metastore_db;create=true</value>  
</property> 
 
<property>  
  <name>javax.jdo.option.ConnectionDriverName</name>  
  <value>org.apache.derby.jdbc.EmbeddedDriver</value>  
</property> 
 
<property>  
  <name>hive.metastore.local</name>  
  <value>true</value>  
</property>  

<property>  
  <name>hive.metastore.warehouse.dir</name>  
  <value>/user/hive/warehouse</value>  
</property>  
   
</configuration> 

启动zookeeper和hadoop

[root@hadoop100 conf]# myself-start-Hahadoop-all.sh 
ZooKeeper JMX enabled by default
Using config: /opt/zookeeper/bin/../conf/zoo.cfg
Starting zookeeper ... already running as process 1508.
ZooKeeper JMX enabled by default
Using config: /opt/zookeeper/bin/../conf/zoo.cfg
Starting zookeeper ... already running as process 1555.
ZooKeeper JMX enabled by default
Using config: /opt/zookeeper/bin/../conf/zoo.cfg
Starting zookeeper ... already running as process 2183.
Starting namenodes on [hadoop100]
hadoop100: starting namenode, logging to /opt/hadoop/logs/hadoop-root-namenode-hadoop100.out
hadoop100: starting datanode, logging to /opt/hadoop/logs/hadoop-root-datanode-hadoop100.out
hadoop102: starting datanode, logging to /opt/hadoop/logs/hadoop-root-datanode-hadoop102.out
hadoop101: starting datanode, logging to /opt/hadoop/logs/hadoop-root-datanode-hadoop101.out
Starting secondary namenodes [hadoop101]
hadoop101: starting secondarynamenode, logging to /opt/hadoop/logs/hadoop-root-secondarynamenode-hadoop101.out
starting yarn daemons
starting resourcemanager, logging to /opt/hadoop/logs/yarn-root-resourcemanager-hadoop100.out
hadoop102: starting nodemanager, logging to /opt/hadoop/logs/yarn-root-nodemanager-hadoop102.out
hadoop100: starting nodemanager, logging to /opt/hadoop/logs/yarn-root-nodemanager-hadoop100.out
hadoop101: starting nodemanager, logging to /opt/hadoop/logs/yarn-root-nodemanager-hadoop101.out
[root@hadoop100 conf]# myself-show-all.sh 
----------------------------- hadoop102 ----------------------------
1508 QuorumPeerMain
1750 NodeManager
1639 DataNode
1885 Jps

----------------------------- hadoop101 ----------------------------
1555 QuorumPeerMain
11764 NodeManager
11594 DataNode
11885 Jps
11662 SecondaryNameNode

----------------------------- hadoop100 ----------------------------
2757 NameNode
3270 NodeManager
2183 QuorumPeerMain
3162 ResourceManager
3595 Jps
2862 DataNode

创建元数据库

[root@hadoop100 conf]# schematool -initSchema -dbType derby
21/09/24 15:31:52 WARN conf.HiveConf: HiveConf of name hive.metastore.local does not exist
Metastore connection URL:	 jdbc:derby:;databaseName=metastore_db;create=true
Metastore Connection Driver :	 org.apache.derby.jdbc.EmbeddedDriver
Metastore connection User:	 APP
Starting metastore schema initialization to 1.2.0
Initialization script hive-schema-1.2.0.derby.sql
Initialization script completed
schemaTool completed

报了个错（高版本不在使用hive.metastore.local，出现了更改一下hive-site就好，不影响使用）

启动hive

[root@hadoop100 conf]# hive

Logging initialized using configuration in jar:file:/opt/hive/lib/hive-common-1.2.1.jar!/hive-log4j.properties
hive> show tables;
OK
Time taken: 0.656 seconds
hive> show databases;
OK
default
Time taken: 0.011 seconds, Fetched: 1 row(s)
hive> 

hive远程安装模式（基于上面的单机模式）

前提：mysql数据库

修改 hive-site

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<!--连接数据的用户名-->
  <property>
      <name>javax.jdo.option.ConnectionUserName</name>
      <value>root</value>
  </property>
<!--连接数据的密码-->
  <property>
      <name>javax.jdo.option.ConnectionPassword</name>
      <value>123456</value>
  </property>
<!--mysql数据库的访问路径，没有路径则自动创建-->
  <property>
      <name>javax.jdo.option.ConnectionURL</name>
      <value>jdbc:mysql://hadoop100:3306/hive?createDatabaseIfNotExist=true&amp;useSSL=false&amp;characterEncoding=UTF-8</value>
  </property>
<!--连接数据库的驱动-->
  <property>
      <name>javax.jdo.option.ConnectionDriverName</name>
      <value>com.mysql.jdbc.Driver</value>
  </property>
<!--元数据是否校验-->

<!--是否自动创建核心文件-->
  <property>
    <name>datanucleus.schema.autoCreateAll</name>
    <value>true</value>
  </property>
<!--thrift服务器绑定的主机-->
  <property>
    <name>hive.server2.thrift.bind.host</name>
    <value>hadoop100</value>
  </property>

<!--默认的存储地址-->

<!--设置显示表头字段名-->
 <property>
   <name>hive.cli.print.header</name>
   <value>true</value>
 </property> 
 <property>
   <name>hive.cli.print.current.db</name>
   <value>true</value>
 </property>
</configuration>

注意：jdbc:mysql://hadoop100:3306/hive?createDatabaseIfNotExist=true&useSSL=false&characterEncoding=UTF-8
;前一定要加&amp！！！！（真实情况可能会添加其他参数）

修改hive-env

export JAVA_HOME=/opt/jdk
export HADOOP_HOME=/opt/hadoop
export HIVE_HOME=/opt/hive
export HIVE_CONF_DIR=/opt/hive/conf

初始化hive

schematool -initSchema -dbType mysql

[root@hadoop100 myself]# schematool -initSchema -dbType mysql
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/hive/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Metastore connection URL:	 jdbc:mysql://hadoop100:3306/hive?createDatabaseIfNotExist=true&useSSL=false;characterEncoding=UTF-8
Metastore Connection Driver :	 com.mysql.jdbc.Driver
Metastore connection User:	 root
org.apache.hadoop.hive.metastore.HiveMetaException: Failed to get schema version.
Underlying cause: java.sql.SQLException : The connection property 'useSSL' only accepts values of the form: 'true', 'false', 'yes' or 'no'. The value 'false;characterEncoding=UTF-8' is not in this set.
SQL Error code: 0
Use --verbose for detailed stacktrace.
*** schemaTool failed ***
[root@hadoop100 myself]# schematool -initSchema -dbType mysql
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/hive/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Metastore connection URL:	 jdbc:mysql://hadoop100:3306/hive?createDatabaseIfNotExist=true&useSSL=false&characterEncoding=UTF-8
Metastore Connection Driver :	 com.mysql.jdbc.Driver
Metastore connection User:	 root
Starting metastore schema initialization to 2.3.0
Initialization script hive-schema-2.3.0.mysql.sql
Initialization script completed
schemaTool completed

启动hive服务

metastore

nohup hive --service metastore  > /dev/null 2>&1 &

[root@hadoop100 myself]# nohup hive --service metastore  > /dev/null 2>&1 &
[1] 6192
[root@hadoop100 myself]# nohup hive --service hiveserver2 > /dev/null 2>&1 &
[2] 6286
[root@hadoop100 myself]# myself-show-all.sh 
----------------------------- hadoop102 ----------------------------
2976 Jps
2899 ResourceManager
2521 DataNode
2618 JournalNode
2414 QuorumPeerMain
2719 NodeManager

----------------------------- hadoop101 ----------------------------
3760 NameNode
3968 DFSZKFailoverController
4099 ResourceManager
4244 NodeManager
3701 QuorumPeerMain
3831 DataNode
4761 Jps
3900 JournalNode

----------------------------- hadoop100 ----------------------------
5152 DataNode
5360 JournalNode
5664 NodeManager
6192 RunJar
5041 NameNode
5553 DFSZKFailoverController
6375 Jps
4889 QuorumPeerMain
6286 RunJar

[root@hadoop100 myself]# 

注意：以上只是弄好了hive服务端！！！！

在其他节点安装hive客户端

复制hive到需要连接的节点

[root@hadoop100 myself]# scp -r /opt/hive/ root@hadoop102:/opt/

修改客户端hive-site

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
        <property>
          <name>hive.metastore.warehouse.dir</name>
          <value>/user/multiuserhive/warehouse</value>
        </property>
        <!--配置是否在本地开启metastore服务, 进程名都是RunJar-->
        <property>
          <name>hive.metastore.local</name>
          <value>false</value>
        </property>
        <!--配置元数据服务的url-->
        <property>
          <name>hive.metastore.uris</name>
           <!-- master是hive服务器的ip -->
          <value>thrift://hadoop100:9083</value> 
        </property>
</configuration>

现在可以使用hadoop102来连接hadoop100的hive服务并操作

[root@hadoop102 conf]# hive
which: no hbase in (/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/opt/jdk/bin:/opt/hadoop/bin:/opt/hadoop/sbin:/opt/zookeeper/bin:/root/bin:/opt/jdk/bin:/opt/hadoop/bin:/opt/hadoop/sbin:/opt/zookeeper/bin:/opt/hive/bin)
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/hive/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]

Logging initialized using configuration in jar:file:/opt/hive/lib/hive-common-2.3.9.jar!/hive-log4j2.properties Async: true
Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
hive> 

当然，你可以弄多个客户端，同样的操作，前提是在本集群中，

在hadoop102使用beeline连接hadoop100的hive服务（jdbc连接）

前提：
添加以下配置到hadoop集群的core-site（关闭集群添加，分发到所有节点）
关闭hive的metastore进程：pkill -f metastore；

			<property>
				<name>hadoop.proxyuser.root.hosts</name>
				<value>*</value>
			</property>
			<property>
				<name>hadoop.proxyuser.root.groups</name>
				<value>*</value>
			</property>

搞定之后
在hive服务端后台启动hiveserver2

 hive --service hiveserver2

在hadoop102连接

[root@hadoop102 ~]# beeline 
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/hive/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Beeline version 2.3.9 by Apache Hive
beeline>  !connect jdbc:hive2://hadoop100:10000
Connecting to jdbc:hive2://hadoop100:10000
Enter username for jdbc:hive2://hadoop100:10000: root
Enter password for jdbc:hive2://hadoop100:10000: ***
Connected to: Apache Hive (version 2.3.9)
Driver: Hive JDBC (version 2.3.9)
Transaction isolation: TRANSACTION_REPEATABLE_READ
0: jdbc:hive2://hadoop100:10000> 

搞定！！！
最后补充
关闭hiveserver2→pkill -f hiveserver2