大数据学习第三天

3. hive的安装

Hive本地模式配置，连接mysql数据库

hive主要是基于hdfs的一个数据仓库，可以实现离线的数据分析任务，原先的hive版本主要是mr为计算引擎，后来随着spark\tez等更高性能的计算引擎的出现，mr开始逐渐被替代。先后出现了hive on tez 和 hive on spark项目，本次先安装hive on hadoop，后面进行整合。

 

3.1 hive的基本安装
1) 解压修改权限
解压hive：

tar -zxvf apache.hive-2.3.3.tar.gz    -C /usr/local/
cd /usr/local/  
mv apache-hive  hive

修改hive的权限和组：

sudo chmod 777 hive
sudo chown -R hadoop hive

2) 配置hive相关的环境变量

vim ~/.bashrc

export HIVE_HOME=/usr/local/hive

export PATH=$HIVE_HOME/bin:$PATH

source ~/.bashrc

3) 修改配置文件

生成配置文件，在conf目录下有四个：

cp hive-env.sh.template hive-env.sh
cp hive-default.xml.template hive-site.xml
cp hive-log4j2.properties.template hive-log4j2.properties
cp hive-exec-log4j2.properties.template hive-exec-log4j2.properties

vim hive-env.sh：

# Set HADOOP_HOME to point to a specific hadoop install directory
HADOOP_HOME=/usr/local/hadoop

# Hive Configuration Directory can be controlled by:
export HIVE_CONF_DIR=/usr/local/hive/conf

# Folder containing extra ibraries required for hive compilation/execution can be controlled by:
export HIVE_AUX_JARS_PATH=/usr/local/hive/lib

vim hive-site.xml：

${system:Java.io.tmpdir}/${hive.session.id}_resources替换为本机路径/tmp/hive/resources 
${system:java.io.tmpdir}/${system:user.name}/operation_logs替换为本机路径/tmp/hive/operation_logs
${system:java.io.tmpdir}/${system:user.name}替换为本机路径 /tmp/hive

 

vim /usr/local/hive/bin/hive-config.sh

export JAVA_HOME=/usr/java/latest

export HIVE_HOME=/usr/local/hive

export HADOOP_HOME=/usr/local/hadoop

在本地创建相应的文件：
cd /tmp
mkdir hive 
mkdir hive/operation_logs 
mkdir hive/resources 
--------------------- 
4) 将默认的derby数据库 替换为mysql

1)安装mysql
yum -y install mysql-server  ：会自动安装mysql-client

service mysqld start

chkconfig mysqld on

使用yum安装mysql connector

yum install -y mysql-connector-java

 

2)将mysql connector拷贝到hive的lib包中

cp /usr/share/java/mysql-connector-java.jar /usr/local/hive/lib

 

3)在mysql上创建hive元数据库，并对hive进行授权

create database if not exists hive_metadata;

grant all privileges on hive_metadata.* to 'hive'@'%' identified by 'hive';

grant all privileges on hive_metadata.* to 'hive'@'localhost' identified by 'hive';

grant all privileges on hive_metadata.* to 'hive'@'spark1' identified by 'hive';

flush privileges;

use hive_metadata;

 

4) 修改hive-site.xml文件，配置mysql的连接属性

vim hive-site.xml只修改下面的内容，其余不用修改

<configuration>
<property>
	<name>hive.metastore.warehouse.dir</name>
	<value>/user/hive/warehouse</value>
	<description>location of default database for the warehouse</description>
</property>
<property>
	<name>hive.exec.scratchdir</name>
	<value>/tmp/hive</value>
	<description>HDFS root scratch dir for Hive jobs which gets created with write all (733) permission. For each connecting user, an HDFS scratch dir: ${hive.exec.scratchdir}/<username> is created, with ${hive.scratch.dir.permission}.</description>
</property>
<property>
	<name>javax.jdo.option.ConnectionURL</name>
	<value>jdbc:mysql://localhost:3306/hive?createDatabaseIfNoExist=true&useSSL=false</value>
	<description> Roy
  JDBC connect string for a JDBC metastore.
  To use SSL to encrypt/authenticate the connection, provide database-specific SSL flag in the connection URL.
  For example, jdbc:postgresql://myhost/db?ssl=true for postgres database.
</description>
</property>
<property>
	<name>javax.jdo.option.ConnectionDriverName</name>
	<value>com.mysql.jdbc.Driver</value>
	<description>User-Defined(Roy) Driver class name for a JDBC metastore</description>
</property>
<property>
	<name>javax.jdo.option.ConnectionUserName</name>
	<value>hive</value>
	<description>User-defined(Roy)Username to use against metastore database</description>
</property>
<property>
	<name>javax.jdo.option.ConnectionPassword</name>
	<value>hive</value>
	<description>User-defined(Roy)password to use against metastore database</description>
</property>
</configuration>

5) 初始化并运行

    初始化mysql

schematool -dbType mysql -initSchema

--------------------- 

启动hadoop： /usr/local/hadoop/sbin/start-all.sh

启动hive：hive

遇到的问题，参考https://blog.csdn.net/l1028386804/article/details/80160042