Hive2.x学习笔记(2)-Hive安装

Hive2.x学习笔记(2)-Hive安装

1. Hive相关资源

1. 官网：https://hive.apache.org
2. 快速入门手册：https://cwiki.apache.org/confluence/display/Hive/GettingStarted
3. 详细手册：https://cwiki.apache.org/confluence/display/Hive/Tutorial
4. HiveQL使用手册：https://cwiki.apache.org/confluence/display/Hive/LanguageManual
5. API手册：https://hive.apache.org/javadoc.html
6. 下载地址：http://www.apache.org/dyn/closer.cgi/hive

2. Hive安装

1. Hive安装
   1. 把apache-hive-2.3.2-bin.tar.gz上传到linux的/opt/目录下
   2. 解压apache-hive-2.3.2-bin.tar.gz到/opt/software目录下面解压apache-hive-2.3.2-bin.tar.gz到/opt/software目录下面
      [hadoop@hadoop100 ~]$ tar -xzf /opt/apache-hive-2.3.2-bin.tar.gz -C /opt/software
   3. 添加软件链接
      [hadoop@hadoop100 ~]$ ln -s /opt/software/apache-hive-2.3.2 hive
   4. 复制hive/conf目录下的hive-env.sh.template为hive-env.sh
      [hadoop@hadoop100 ~]$ cp hive-env.sh.template hive-env.sh
   5. 配置hive-env.sh如下

# Hive Client memory usage can be an issue if a large number of clients
# are running at the same time. The flags below have been useful in
# reducing memory usage:
#
# if [ "$SERVICE" = "cli" ]; then
#   if [ -z "$DEBUG" ]; then
#     export HADOOP_OPTS="$HADOOP_OPTS -XX:NewRatio=12 -Xms10m -XX:MaxHeapFreeRatio=40 -XX:MinHeapFreeRatio=15 -XX:+UseParNewGC -XX:-UseGCOverheadLimit"
#   else
#     export HADOOP_OPTS="$HADOOP_OPTS -XX:NewRatio=12 -Xms10m -XX:MaxHeapFreeRatio=40 -XX:MinHeapFreeRatio=15 -XX:-UseGCOverheadLimit"
#   fi
# fi

# The heap size of the jvm stared by hive shell script can be controlled via:
#
# export HADOOP_HEAPSIZE=1024
#
# Larger heap size may be required when running queries over large number of files or partitions.
# By default hive shell scripts use a heap size of 256 (MB).  Larger heap size would also be
# appropriate for hive server.


# Set HADOOP_HOME to point to a specific hadoop install directory
# 配置HADOOP的安装路径
HADOOP_HOME=/opt/software/hadoop-2.9.1

# Hive Configuration Directory can be controlled by:
# 配置Hive的配置文件路径
HIVE_CONF_DIR=/opt/software/apache-hive-2.3.2-bin/conf
# Folder containing extra libraries required for hive compilation/execution can be controlled by:
# export HIVE_AUX_JARS_PATH=

6. 修改/etc/profile环境变量，在文件未尾修改如下

export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.191.b12-1.el7_6.x86_64/
export HADOOP_HOME=/opt/module/hadoop-2.7.7
export HIVE_HOME=/opt/module/apache-hive-2.3.4-bin
export PATH=$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$HIVE_HOME/bin:$PATH

3. Hadoop集群配置
   1. 启动hdfs和yarn

[hadoop@hadoop100 ~]$ start-dfs.sh
[hadoop@hadoop100 ~]$ start-yarn.sh

2. 在hdfs上创建/tmp和/user/hive/warehouse两个目录，并给同组权限添加上可写

[hadoop@hadoop100 ~]$ hadoop fs -mkdir /tmp
[hadoop@hadoop100 ~]$ hadoop fs -mkdir -p /user/hive/warehouse
[hadoop@hadoop100 ~]$ hadoop fs -chmod g+w /tmp
[hadoop@hadoop100 ~]$ hadoop fs -chmod g+w /user/hive/warehouse

4. Hive元数据配置到MySQL
   Hive默认使用derby做为默认数据库
   1. 拷贝mysql驱动到hive/lib
      [hadoop@hadoop100 ~]$ copy mysql-connector-java-5.1.47.jar hive/lib
   2. 在hive/conf目录下创建hive-site.xml
      [hadoop@hadoop100 ~]$ touch hive-site.xml
   3. 参考官方文档配置参数
      https://cwiki.apache.org/confluence/display/Hive/AdminManual+Metastore+Administration

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
  <property>
    <name>javax.jdo.option.ConnectionURL</name>
    <value>jdbc:mysql://192.168.248.101/metastore?createDatabaseIfNotExist=true</value>
  </property>
  <property>
    <name>javax.jdo.option.ConnectionDriverName</name>
    <value>com.mysql.jdbc.Driver</value>
  </property>
  <property>
    <name>javax.jdo.option.ConnectionUserName</name>
    <value>root</value>
  </property>
  <property>
    <name>javax.jdo.option.ConnectionPassword</name>
    <value>root</value>
  </property>
</configuration>

5. 启动hive
   1. 初始化schema
      schematool -dbType mysql -initSchema
   2. 测试hive

[hadoop@hadoop100 ~]$ hive

hive> show databases;
OK
default

至此，Hive安装成功。

3. Hive常见属性配置

1. 数据仓库位置配置
   数据仓库默认保存在hdfs的/user/hive/warehouse路径下。在创建目录下，没有对默认数据库default创建目录。如果某张表属于default数据库，直接在数据仓库目录下创建一个目录。
   可能通过修改hive-sixe.xml，指定数据仓库的位置。记住需要给目录赋写权限。

<property>
  <name>hive.metastore.warehouse.dir</name>
  <value>/user/hive/warehouse</value>
</property>

hdfs dfs -chmod g+w /user/hive/warehouse

2. 查询显示表头信息配置
   在hive-site.xml添加如下配置，就可以实现显示表头

<property>
  <name>hive.cli.print.header</name>
  <value>true</value>
</property>
<property>
  <name>hive.cli.print.db</name>
  <value>true</value>
</property>

修改后结果

3. 运行日志信息配置
   Hive的log默认存放在用户主目录下，通过以下步骤可以修改日志的存放位置
   1. 复制hive/conf/hive-log4j.properties.template为hive/conf/hive-log4j.properties
   2. 在hive-log4j.properties文件中修改log存位位置
      hive.log.dir=hive/logs