Part I - 文件清单
- Hive安装包 apache-hive-1.2.2-bin.tar.gz
- HIve-HWI Web监测:apache-hive-1.2.2-src.tar.gz
- HIve-HWI Web编译依赖环境:apache-ant-1.10.7-bin.tar.gz
- Hive与MySQ链接包:mysql-connector-java-5.1.39.jar
- 避免HWI出现JSP问题:jasper-compiler-5.5.23.jar
- 避免HWI出现JSP问题:jasper-runtime-5.5.23.jar
- 避免HWI出现JSP问题:commons-el-5.5.23.jar
Part II - 下载链接
[hive -src或者bin文件 下载链接]: https://hive.apache.org/downloads.html
[ant 下载链接]:https://ant.apache.org/bindownload.cgi
[jasper-compiler-5.5.23、jasper-runtime-5.5.23.jar和commons-el-5.5.23.jar]:https://mvnrepository.com/
Part III - Hive分布式部署
总览,集群信息:
节点角色 | Master | Slave1 | Slave2 |
---|---|---|---|
IP | 192.168.137.128 | 192.168.137.129 | 192.168.137.130 |
HostName | BlogMaster | BlogSlave1 | BlogSlave2 |
Hadoop | BolgMaster-YES | BlogSlave1-YES | BlogSlave2-YES |
Zookeeper | BolgMaster-YES | BlogSlave1-YES | BlogSlave2-YES |
Hive | BolgMaster-YES | BlogSlave1-NO | BlogSlave2-NO |
MySQL | BolgMaster-YES | BlogSlave1-NO(可选) | BlogSlave2-NO(可选) |
Step 1: Hive依赖包配置-Ant并配置环境变量
解压ant至指定的目录,
[root@BlogMaster ~]# tar -zxvf apache-ant-1.10.7-bin.tar.gz -C /opt/cluster/
并进入profile文件,进行环境变量的配置,具体添加如下内容:
export ANT_HOME=/opt/cluster/apache-ant-1.10.7
export PATH=$PATH:$JAVA_HOME/bin:$JAVA_HOME/jre:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$HIVE_HOME/bin:$ANT_HOME/bin
后执行source profile,使得环境变量配置生效。
Step 2: Hive部署
Hive仅需在BlogMaster节点安装。
- Step 2.1: 解压hive文件至指定目录(/opt/cluster)
此处执行如下命令
[root@BlogMaster ~]# tar -zxvf apache-hive-1.2.2-bin.tar.gz -C /opt/cluster/
- Step 2.2: 配置hive-env.sh文件(位于/opt/cluster/apache-hive-1.2.2-bin/conf)
进入该文件后,添加如下内容:
export JAVA_HOME=/opt/cluster/jdk1.8.0_181
export HIVE_HOME=/opt/cluster/apache-hive-1.2.2-bin
# Set HADOOP_HOME to point to a specific hadoop install directory
HADOOP_HOME=/opt/cluster/hadoop-2.8.4
# Hive Configuration Directory can be controlled by:
export HIVE_CONF_DIR=/opt/cluster/apache-hive-1.2.2-bin/conf
# Folder containing extra ibraries required for hive compilation/execution can be controlled by:
export HIVE_AUX_JARS_PATH=/opt/cluster/apache-hive-1.2.2-bin/lib
- Step 2.3: 配置hive-site.xml文件(位于/opt/cluster/apache-hive-1.2.2-bin/conf)
若没有该文件,直接创建一个hive-site.xml的文件,并添加如下内容
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
<name>javax.jdo.option.ConnectionURL</name>
<value>jdbc:mysql://BlogMaster:3306/metastore?createDatabaseIfNotExist=true</value>
<description>JDBC connect string for a JDBC metastore</description>
</property>
<property>
<name>javax.jdo.option.ConnectionDriverName</name>
<value>com.mysql.jdbc.Driver</value>
<description>Driver class name for a JDBC metastore</description>
</property>
<property>
<name>javax.jdo.option.ConnectionUserName</name>
<value>root</value>
<description>username to use against metastore database</description>
</property>
<property>
<name>javax.jdo.option.ConnectionPassword</name>
<value>201012</value>
<description>password to use against metastore database</description>
</property>
<property>
<name>hive.cli.print.header</name>
<value>true</value>
</property>
<property>
<name>hive.cli.print.current.db</name>
<value>true</value>
</property>
<property>
<name>hive.hwi.listen.host</name>
<value>0.0.0.0</value>
</property>
<property>
<name>hive.hwi.listen.port</name>
<value>9999</value>
</property>
<property>
<name>hive.hwi.war.file</name>
<value>lib/hive-hwi-1.2.2.war</value>
</property>
</configuration>
- Step 2.4: 编译hive源码,获取HWI以来的web包
第一步: 解压apache-hive-1.2.2-src.tar.gz,具体执行如下命令
[root@BlogMaster ~]# tar -zxvf apache-hive-1.2.2-src.tar.gz
第二步: 进入该文件解压目录下的hwi,进行如下编译命令
[root@BlogMaster hwi]# jar cvfM0 hive-hwi-1.2.2.war -C web/ .
此时,若出现如下结果,则说明编译成功。
[root@BlogMaster hwi]# jar cvfM0 hive-hwi-1.2.2.war -C web/ .
adding: css/(in = 0) (out= 0)(stored 0%)
adding: css/bootstrap.min.css(in = 90193) (out= 90193)(stored 0%)
adding: img/(in = 0) (out= 0)(stored 0%)
adding: img/glyphicons-halflings-white.png(in = 4352) (out= 4352)(stored 0%)
adding: img/glyphicons-halflings.png(in = 4352) (out= 4352)(stored 0%)
adding: WEB-INF/(in = 0) (out= 0)(stored 0%)
adding: WEB-INF/web.xml(in = 1438) (out= 1438)(stored 0%)
adding: authorize.jsp(in = 2729) (out= 2729)(stored 0%)
adding: diagnostics.jsp(in = 2365) (out= 2365)(stored 0%)
adding: error_page.jsp(in = 1867) (out= 1867)(stored 0%)
adding: index.jsp(in = 1876) (out= 1876)(stored 0%)
adding: left_navigation.jsp(in = 1553) (out= 1553)(stored 0%)
adding: navbar.jsp(in = 1345) (out= 1345)(stored 0%)
adding: session_create.jsp(in = 2690) (out= 2690)(stored 0%)
adding: session_diagnostics.jsp(in = 2489) (out= 2489)(stored 0%)
adding: session_history.jsp(in = 3150) (out= 3150)(stored 0%)
adding: session_kill.jsp(in = 2236) (out= 2236)(stored 0%)
adding: session_list.jsp(in = 2298) (out= 2298)(stored 0%)
adding: session_manage.jsp(in = 6738) (out= 6738)(stored 0%)
adding: session_remove.jsp(in = 2359) (out= 2359)(stored 0%)
adding: session_result.jsp(in = 2488) (out= 2488)(stored 0%)
adding: show_database.jsp(in = 2346) (out= 2346)(stored 0%)
adding: show_databases.jsp(in = 2096) (out= 2096)(stored 0%)
adding: show_table.jsp(in = 4996) (out= 4996)(stored 0%)
adding: view_file.jsp(in = 2653) (out= 2653)(stored 0%)
第三步: 拷贝新生成的war包至hive安装目录下的lib子目录
[root@BlogMaster hwi]# cp hive-hwi-1.2.2.war /opt/cluster/apache-hive-1.2.2-bin/lib/
- Step 2.5: 配置hive环境变量
进入/etc下的profile文件,新添加对应HIVE_HOME即内部的bin目录,具体如下:
export HIVE_HOME=/opt/cluster/apache-hive-1.2.2-bin
export PATH=$PATH:$JAVA_HOME/bin:$JAVA_HOME/jre:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$HIVE_HOME/bin:$ANT_HOME/bin
之后执行source profile,确保配置生效。
- Step 2.6: 新增三个jar包至hive安装目录下的lib子目录
为避免出现加载Hive HWI页面时,出现JSP问题,需要将如下三个包拷贝至hive安装目录的lib子目录下。
1. jasper-compiler-5.5.23.jar
2. jasper-runtime-5.5.23.jar
3. commons-el-5.5.23.jar
- Step 2.7: 新增java的tools.jar包至hive安装目录下的lib子目录
[root@BlogMaster ~]# cp /opt/cluster/jdk1.8.0_181/lib/tools.jar /opt/cluster/apache-hive-1.2.2-bin/lib/
- Step 2.8: 新增与MySQL连接的jar包至hive安装目录下的lib子目录
[root@BlogMaster ~]# cp mysql-connector-java-5.1.39.jar /opt/cluster/apache-hive-1.2.2-bin/lib
- Step 2.9: 进入Hive安装目录下conf目录,修改hive-log4j.properties文件
进入该目录后,可能并没有hive-log4j.properties文件,但会有一个相似的hive-log4j.properties.template文件,这里采用cp命令拷贝该文件并重命名为hive-log4j.properties。之后进入hive-log4j.properties,修改其hive.log.dir选项,具体如下:
hive.log.dir=/opt/cluster/apache-hive-1.2.2-bin/logs
- Step 2.10: 进入Hive安装目录下创建一个名为“logs”的文件夹,用于存放hive运行状态监测的日志记录
[root@BlogMaster apache-hive-1.2.2-bin]# mkdir logs
Step 3: MySQL配置,以免只能hive单窗口操作
MySQL节点可在任意节点安装,但需注意设置其与Hive交互的IP地址的更改。为方便操作,这里MySQL也将部署与BlogMaster节点。 在进行如下操作时,须确保主节点并未配置mysql,若自带,务必直接删除。
判断主节点BlogMaster是否包含MySQL,可执行命令:yum list installed | grep mysql。
若包含,则执行yum -y remove +数据库名称(此处数据库为上述命令执行后出现的MySQL数据库)。
- Step 3.1: 执行yum install libaio命令,安装依赖环境
[root@BlogMaster ~]# yum install libaio
- Step 3.2: 执行yum install wget
[root@BlogMaster ~]# yum install wget
- Step 3.3: 执行wget http://dev.mysql.com/get/mysql-community-release-el7-5.noarch.rpm
[root@BlogMaster ~]# wget http://dev.mysql.com/get/mysql-community-release-el7-5.noarch.rpm
- Step 3.4: 添加 MySQL Yum Repository 到你的系统 repository 列表中,执行
[root@BlogMaster ~]# yum localinstall mysql-community-release-el7-5.noarch.rpm
- Step 3.5: 验证下是否添加成功
[root@BlogMaster ~]# yum repolist enabled | grep "mysql.*-community.*"
- Step 3.6: 验证下是否添加成功
[root@BlogMaster ~]# yum repolist enabled | grep "mysql.*-community.*"
出现如下结果,则说明添加成功。
[root@BlogMaster ~]# yum repolist enabled | grep "mysql.*-community.*"
mysql-connectors-community/x86_64 MySQL Connectors Community 131
mysql-tools-community/x86_64 MySQL Tools Community 100
mysql56-community/x86_64 MySQL 5.6 Community Server 496
- Step 3.7: 查看 MySQL 版本,执行
[root@BlogMaster ~]# yum repolist all | grep mysql
[root@BlogMaster ~]# yum repolist enabled | grep mysql
- Step 3.8: 通过 Yum 来安装 MySQL
[root@BlogMaster ~]# yum install mysql-community-server
此处较长时间的等待,可以听一首徐佳莹的失落沙洲,enjoy…
最后,中间出现提示请选择"y"一路回车。若最终出现如下结果,则说明安装成功,即提示包含了 mysql-community-server、mysql-community-client、mysql-community-common、mysql-community-libs 四个包。
- Step 3.9: 执行如下命令
[root@BlogMaster ~]# rpm -qi mysql-community-server.x86_64 0:5.6.24-3.el7
- Step 3.10: 启动MySQL服务
[root@BlogMaster ~]# systemctl start mysqld
- Step 3.11: 远程访问 MySQL, 需开放默认端口号 3306.
[root@BlogMaster ~]# firewall-cmd --permanent --zone=public --add-port=3306/tcp
[root@BlogMaster ~]# firewall-cmd --permanent --zone=public --add-port=3306/udp
[root@BlogMaster ~]# firewall-cmd --reload
- Step 3.12: MySQL 安全设置,配置MySQLroot用户的密码
[root@BlogMaster ~]# mysql_secure_installation
中间过程中,出现提示直接输入“y”;另外出现设置根用户密码的提示时,设置自己的密码,但请务必与hive-site中设置与MySQL连接的密码一致,即2.3小节的
<property>
<name>javax.jdo.option.ConnectionURL</name>
<value>jdbc:mysql://BlogMaster:3306/metastore?createDatabaseIfNotExist=true</value>
<description>JDBC connect string for a JDBC metastore</description>
</property>
<property>
<name>javax.jdo.option.ConnectionDriverName</name>
<value>com.mysql.jdbc.Driver</value>
<description>Driver class name for a JDBC metastore</description>
</property>
<property>
<name>javax.jdo.option.ConnectionUserName</name>
<value>root</value>
<description>username to use against metastore database</description>
</property>
<property>
<name>javax.jdo.option.ConnectionPassword</name>
<value>201012</value>
<description>password to use against metastore database</description>
</property>
- Step 3.13: MySQL 安全设置,配置MySQLroot用户的密码
第一步:进入MySQL shell,其中用户为root,密码为自己设置的密码(这里设置的201012)。
[root@BlogMaster ~]# mysql -uroot -p201012
Warning: Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 64
Server version: 5.6.46 MySQL Community Server (GPL)
Copyright (c) 2000, 2019, Oracle and/or its affiliates. All rights reserved.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
第二步:进入后,选择名称为mysql的数据库,
mysql> show databases;
+--------------------+
| Database |
+--------------------+
| information_schema |
| metastore |
| mysql |
| performance_schema |
+--------------------+
4 rows in set (0.11 sec)
mysql> use mysql;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A
Database changed
mysql>
值得注意,这里出现的metastore即为hive与MySQL交互的hive的元数据。
第三步:查询user表
第四步:修改 user表,把 Host表内容修改为 %
mysql>update user set host='%' where host='localhost';
第五步:删除root用户的其它host
mysql>delete from user where Host='127.0.0.1';
mysql>delete from user where Host='::1';
第六步:刷新,使设置生效
mysql>flush privileges;
之后执行quit命令,退出MySQL即可。
Part IV - Hive运行状态监测
Step 1: 启动hive之间,务必启动Hadoop集群和Zookeeper服务
在按照此前博客启动Hadoop集群的基础上,执行hive --service hwi命令。
[root@BlogMaster ~]# hive --service hwi
出现如下结果,则说明监测成功。
19/11/13 17:15:59 INFO hwi.HWIServer: HWI is starting up
19/11/13 17:16:10 INFO mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
19/11/13 17:16:10 INFO mortbay.log: jetty-6.1.26
19/11/13 17:16:10 INFO mortbay.log: Extract /opt/cluster/apache-hive-1.2.2-bin/lib/hive-hwi-1.2.2.war to /tmp/Jetty_0_0_0_0_9999_hive.hwi.1.2.2.war__hwi__21w1ka/webapp
19/11/13 17:16:11 INFO mortbay.log: Started SocketConnector@0.0.0.0:9999
Step 2: 执行如下命令查看Hive情况
- Step 2.1: Jps查看
[root@BlogMaster lib]# jps
7746 NameNode
8291 QuorumPeerMain
15255 Jps
8029 ResourceManager
15182 RunJar
出现RunJar进程,则说明启动成功。
- Step 2.2:Hive Web页面查看
查看Hive与HDFS交互的数据,登录网址:http://192.168.137.128:50070/explorer.html#/user/hive/warehouse
直接进入Hive Web页面,查看创建的student_db数据库。登录网址为: http://192.168.137.128:9999/hwi