HIVE_第一讲_ HIVE入门 , 搭建 单节点的基于内置的 derby 数据库

版权声明:本文为博主原创文章,未经博主允许不得转载。 https://blog.csdn.net/u010003835/article/details/80316056

最近博主希望优化现有的业务框架,打算搭建一套 以Hive 为存储的数据仓库系统。

之前没有了解过hive , 本次是首次使用Hive, 不足之处欢迎各位指正。


Tips : 

在安装HIVE 之前请确保,已经安装了Hadoop


此种方式存在不足:不能使用多个链接 连接Hive 数据库,会报错!!!!!!

解决方法: 使用外部的元数据库,可以参考我接下来的一篇文章。

HIVE_第二讲_ Hive on MySQL


第一步:下载压缩包

首先我们需要从 Hive 官网上下载一个最新的 tar.gz 的压缩包


我下载的版本是 apache-hive-2.3.3-bin

http://mirrors.shu.edu.cn/apache/hive/hive-2.3.3/



第二步:  解压

下载好之后 ,我们对压缩包进行解压

解压指令

    tar -zxvf xx.tar.gz -C outputDir  



第三步: 修改配置文件


首先我们查看下conf目录,我们需要对其中的配置文件进行修改。


这里对其中2个文件进行讲解:

hive-default.xml.template

hive-env.sh.template

hive-default.xml.template  

   里面写有所有hive 可配置属性的默认设置与解释,如果我们忘了某一个配置的含义,可以来这里面进行查找。

hive-env.sh.template

   描述了hive 所需的环境变量的相关设置


复制一份hive-env.sh 填入以下内容:

hive-env.sh

# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements.  See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership.  The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License.  You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

# Set Hive and Hadoop environment variables here. These variables can be used
# to control the execution of Hive. It should be used by admins to configure
# the Hive installation (so that users do not have to set environment variables
# or set command line parameters to get correct behavior).
#
# The hive service being invoked (CLI etc.) is available via the environment
# variable SERVICE

# Hive Client memory usage can be an issue if a large number of clients
# are running at the same time. The flags below have been useful in 
# reducing memory usage:
#
# if [ "$SERVICE" = "cli" ]; then
#   if [ -z "$DEBUG" ]; then
#     export HADOOP_OPTS="$HADOOP_OPTS -XX:NewRatio=12 -Xms10m -XX:MaxHeapFreeRatio=40 -XX:MinHeapFreeRatio=15 -XX:+UseParNewGC -XX:-UseGCOverheadLimit"
#   else
#     export HADOOP_OPTS="$HADOOP_OPTS -XX:NewRatio=12 -Xms10m -XX:MaxHeapFreeRatio=40 -XX:MinHeapFreeRatio=15 -XX:-UseGCOverheadLimit"
#   fi
# fi

# The heap size of the jvm stared by hive shell script can be controlled via:
#
export HADOOP_HEAPSIZE=1024
#
# Larger heap size may be required when running queries over large number of files or partitions. 
# By default hive shell scripts use a heap size of 256 (MB).  Larger heap size would also be 
# appropriate for hive server.


# Set HADOOP_HOME to point to a specific hadoop install directory
HADOOP_HOME=/usr/local/hadoop

# Hive Configuration Directory can be controlled by:
export HIVE_CONF_DIR=/usr/local/hive/conf

# Folder containing extra libraries required for hive compilation/execution can be controlled by:
export HIVE_AUX_JARS_PATH=/usr/local/hive/lib

我们新创建一个hive-site.xml , 填写一些我们自己定义的配置:

hive-site.xml

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
	<property>
		<name>hive.metastore.warehouse.dir</name>
		<value>/usr/local/hive_home/apache-hive-2.3.3-bin/my-warehouse</value>
		<description>
			Local or HDFS directory where Hive keeps table contents.
		</description>
	</property>
	<property>
                <name>hive.metastore.local</name>
                <value>true</value>
                <description>
			Use false if a production metastore server is used.
                </description>
        </property>
	<property>
                <name>javax.jdo.option.ConnectionURL</name>
                <value>jdbc:derby:;databaseName=/usr/local/hive_home/apache-hive-2.3.3-bin/my-metastore;create=true</value>
                <description>
			The JDBC connection URL.
                </description>
        </property>
</configuration>


对其中的重要属性进行讲解:

hive.metastore.warehouse.dir  

 描述了spark 中管理表 存放的默认位置。其中 default 库 会存放在根目录下,而其他数据库的表 会存放在 以数据库名开头的 一个子目录下 database/tablename/


hive.metastore.local

 描述了hive 元信息的存放位置,元信息指的是 对hive 中表的描述,存放位置。。等相关信息。


修改完这两个hive 之后我们的配置文件已经配置完成,


第四步.  初始化derby 数据库

对于第一次配置derby,此时的derby 数据库还未完成初始化,我们需要完成初始化服务。

运行 bin 下面的schematool 工具

schematool -dbType derby -initSchema



第五步.  启动hive服务


启动hive服务,bin/hive 

( 默认启动的服务为 hive-cli, 可以通过 --help 进行查找帮助)



我们也可以通过 bin/hive --service cli --help  更具体的cli 服务 的帮助文档



我们尝试操作下 看下所有的数据库集合 show databases 


[root@master hive]# bin/hive --service cli 
which: no hbase in (/usr/local/flume/bin:/usr/local/redis/redis-4.0.1/bin:/usr/local/storm/apache-storm-1.0.1/bin:/usr/local/spark/bin:/usr/local/hadoop/bin:/usr/local/hadoop/sbin:/usr/local/scala_home/scala-2.12.4/bin:/usr/local/java_home/jdk1.8.0_162/bin:/usr/lib64/qt-3.3/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/git/bin/::/usr/local/nginx/sbin:/usr/local/zookeeper/zookeeper-3.4.5/bin/:/usr/local/kafka/kafka_2.11-0.10.0.1/bin/:/usr/local/kafka/:/usr/local/hive/bin:/usr/local/mongodb/mongodb-linux-x86_64-rhel62-3.2.9/bin/:/usr/local/cassandra/apache-cassandra-2.1.15/bin/:/root/bin)
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/local/hive_home/apache-hive-2.3.3-bin/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/local/hadoop_home/hadoop-2.9.0/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]

Logging initialized using configuration in file:/usr/local/hive_home/apache-hive-2.3.3-bin/conf/hive-log4j2.properties Async: true
Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
hive (default)> show databases;
OK
default
Time taken: 4.938 seconds, Fetched: 1 row(s)


大功告成!!

阅读更多
想对作者说点什么?

博主推荐

换一批

没有更多推荐了,返回首页