环境介绍:hadoop-2.9.1、hive-1.2.0
节点规划:
mini0 hadoop hive(server) metastore
mini1 hadoop hive(client) mysql
mini2 hadoop hive(client)
1、安装MYSQL
CentOS7 64位下MySQL5.7安装与配置:http://www.linuxidc.com/Linux/2016-09/135288.htm
mysql-connector-java.jar下载:https://www.cnblogs.com/jizuiku/p/7637125.html
2、配置MYSQL Service and Connector
说明:因为使用MySQL作为存储元数据的数据库,所以需要把连接MySQL的jar包放入或链接到$HIVE_HOME/lib目录下。
3、
3.1、mysql配置
启动MySQL服务:systemctl start mysqld
查看MySQL状态:systemctl status mysqld
开机启动:systemctl enable mysqld
;systemctl daemon-reload
重启Mysql服务:service mysqld restart
或者 systemctl restart mysqld.service
默认只允许root帐户在本地登录,如果要在其它机器上连接mysql,必须修改root允许远程连接;或者添加一个允许远程连接的帐户,为了安全起见,我添加一个新的帐户:
创建数据库
drop database metastore;
CREATE DATABASE metastore;
USE metastore;
SOURCE /home/hadoop/apps/hive/scripts/metastore/upgrade/mysql/hive-schema-0.9.0.mysql.sql;
创建用户
CREATE USER 'hive'@'mini0' IDENTIFIED BY 'zys123';
REVOKE ALL PRIVILEGES, GRANT OPTION FROM 'hive'@'mini0';
GRANT ALL ON metastore.* TO 'hive'@'mini0' IDENTIFIED BY 'zys123';
GRANT ALL ON metastore.* TO 'hive'@'%' IDENTIFIED BY 'zys123';
FLUSH PRIVILEGES;
ALTER DATABASE metastore CHARACTER SET latin1;
select user,host from mysql.user;
flush privileges;
quit;
删除用户
select user,host from mysql.user;
delete from mysql.user where user='hive' and host='%';
delete from mysql.user where user='hive' and host='mini0';
drop user 'hive'@'%';
drop user 'hive'@'mini0';
flush privileges;
4.配置hive
4.1、HDFS存储位置配置(Hive配置文件里要用到HDFS的一些路径,需要先手动创建。)
hdfs dfs -mkdir -p /usr/hive/warehouse
hdfs dfs -mkdir -p /usr/hive/tmp
hdfs dfs -mkdir -p /usr/hive/log
hdfs dfs -chmod 777 /usr/hive/warehouse
hdfs dfs -chmod 777 /usr/hive/tmp
hdfs dfs -chmod 777 /usr/hive/log
上述语句涉及hive-site.xml hive.metastore.warehouse.dir等,表示数据在hdfs中的存储位置
4.2 配置hive-env.sh
export HIVE_CONF_DIR=/home/hadoop/apps/hive/conf
export HADOOP_HOME=/home/hadoop/apps/hadoop-2.9.1
export HIVE_HOME=/home/hadoop/apps/hive
然后配置hive-log4j.properties
hive.log.dir=/usr/lib/hive/logs
4.3、服务端hive-site.xml(mini0)
(服务端指的是Metastore服务所在的机器,即安装metastore的机器)
<configuration>
<property>
<name>javax.jdo.option.ConnectionURL</name>
<value>jdbc:mysql://mini1:3306/metastore?createDatabaseIfNotExist=true</value>
<description>JDBC connect string for a JDBC metastore</description>
</property>
<property>
<name>javax.jdo.option.ConnectionDriverName</name>
<value>com.mysql.jdbc.Driver</value>
<description>Driver class name for a JDBC metastore</description>
</property>
<property>
<name>javax.jdo.option.ConnectionUserName</name>
<value>hive</value>
<description>username to use against metastore database</description>
</property>
<property>
<name>javax.jdo.option.ConnectionPassword</name>
<value>zys123</value>
<description>password to use against metastore database</description>
</property>
<property>
<name>hive.metastore.warehouse.dir</name>
<value>/usr/hive/warehouse</value>
</property>
<property>
<name>hive.exec.scratchdir</name>
<value>/usr/hive/tmp</value>
</property>
<property>
<name>hive.querylog.location</name>
<value>/usr/hive/log</value>
</property>
</configuration>
1、客户端hive-site.xml(mini1、mini2)
<configuration>
<property>
<name>hive.metastore.uris</name>
<value>thrift://mini0:9083</value>
<description>IP address (or fully-qualified domain name) and port of the metastore host</description>
</property>
<property>
<name>hive.metastore.warehouse.dir</name>
<value>/usr/hive/warehouse</value>
</property>
<property>
<name>hive.exec.scratchdir</name>
<value>/usr/hive/tmp</value>
</property>
<property>
<name>hive.querylog.location</name>
<value>/usr/hive/log</value>
</property>
</configuration>
6、Jline包版本不一致的问题,需要拷贝hive的lib目录中jline.2.12.jar的jar包替换掉hadoop中的
/home/hadoop/app/hadoop-2.9.1/share/hadoop/yarn/lib/jline-0.9.94.jar
7、metastore初始化(必须要初始化,否则后边hiveserver2无法启动):
schematool -dbType mysql -initSchema
8、启动Hive
1、**本地方式启动**hive:hive
2、
服务端启动:通过Hive thrift服务(远程)
启动为前台:hiveserver2
启动为后台:nohup hiveserver2 &
客户端启动:
方式一:beeline回车,进入beeline的命令界面
输入命令连接:!connect jdbc:hive2//mini0:10000 -n hadoop
(mini0是hiveserver2所启动的那台主机名,端口默认是10000)
方式二:直接启动:beeline -u jdbc:hive2://mini0:10000 -n hadoop
查看hiveserver2服务端状态:netstat -nltp | grep 10000
Hive基本操作:
1、hive命令:
清空表:truncate table person;
创建数据库:create database test_db
删除数据库:drop database test_db;
删除表:drop table ****;
6.建表(默认是内部表)
create table person(name string,id int) row format delimited fields terminated by '\t';
建分区表
create table td_part(id bigint, account string, income double, expenses double, time string) partitioned by (logdate string) row format delimited fields terminated by '\t';
建外部表
create external table td_ext(id bigint, account string, income double, expenses double, time string) row format delimited fields terminated by '\t' location '/td_ext';
参考网站: Hive安装配置指北(含Hive
Metastore详解):https://www.cnblogs.com/linbingdong/p/5829369.html