大数据平台分布式搭建 - Hive(HWI)+MySQL分布式配置

Part I - 文件清单
  1. Hive安装包 apache-hive-1.2.2-bin.tar.gz
  2. HIve-HWI Web监测:apache-hive-1.2.2-src.tar.gz
  3. HIve-HWI Web编译依赖环境:apache-ant-1.10.7-bin.tar.gz
  4. Hive与MySQ链接包:mysql-connector-java-5.1.39.jar
  5. 避免HWI出现JSP问题:jasper-compiler-5.5.23.jar
  6. 避免HWI出现JSP问题:jasper-runtime-5.5.23.jar
  7. 避免HWI出现JSP问题:commons-el-5.5.23.jar
Part II - 下载链接

[hive -src或者bin文件 下载链接]: https://hive.apache.org/downloads.html
[ant 下载链接]:https://ant.apache.org/bindownload.cgi
[jasper-compiler-5.5.23、jasper-runtime-5.5.23.jar和commons-el-5.5.23.jar]:https://mvnrepository.com/

Part III - Hive分布式部署

总览,集群信息:

节点角色MasterSlave1Slave2
IP192.168.137.128192.168.137.129192.168.137.130
HostNameBlogMasterBlogSlave1BlogSlave2
HadoopBolgMaster-YESBlogSlave1-YESBlogSlave2-YES
ZookeeperBolgMaster-YESBlogSlave1-YESBlogSlave2-YES
HiveBolgMaster-YESBlogSlave1-NOBlogSlave2-NO
MySQLBolgMaster-YESBlogSlave1-NO(可选)BlogSlave2-NO(可选)

Step 1: Hive依赖包配置-Ant并配置环境变量

解压ant至指定的目录,

[root@BlogMaster ~]# tar -zxvf apache-ant-1.10.7-bin.tar.gz -C /opt/cluster/

并进入profile文件,进行环境变量的配置,具体添加如下内容:

export ANT_HOME=/opt/cluster/apache-ant-1.10.7
export PATH=$PATH:$JAVA_HOME/bin:$JAVA_HOME/jre:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$HIVE_HOME/bin:$ANT_HOME/bin

后执行source profile,使得环境变量配置生效。

Step 2: Hive部署

Hive仅需在BlogMaster节点安装。

  • Step 2.1: 解压hive文件至指定目录(/opt/cluster)

此处执行如下命令

[root@BlogMaster ~]# tar -zxvf apache-hive-1.2.2-bin.tar.gz -C /opt/cluster/
  • Step 2.2: 配置hive-env.sh文件(位于/opt/cluster/apache-hive-1.2.2-bin/conf)

进入该文件后,添加如下内容:

export JAVA_HOME=/opt/cluster/jdk1.8.0_181
export HIVE_HOME=/opt/cluster/apache-hive-1.2.2-bin

# Set HADOOP_HOME to point to a specific hadoop install directory
HADOOP_HOME=/opt/cluster/hadoop-2.8.4

# Hive Configuration Directory can be controlled by:
export HIVE_CONF_DIR=/opt/cluster/apache-hive-1.2.2-bin/conf

# Folder containing extra ibraries required for hive compilation/execution can be controlled by:
export HIVE_AUX_JARS_PATH=/opt/cluster/apache-hive-1.2.2-bin/lib
  • Step 2.3: 配置hive-site.xml文件(位于/opt/cluster/apache-hive-1.2.2-bin/conf)

若没有该文件,直接创建一个hive-site.xml的文件,并添加如下内容

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
        <property>
                <name>javax.jdo.option.ConnectionURL</name>
                <value>jdbc:mysql://BlogMaster:3306/metastore?createDatabaseIfNotExist=true</value>
                <description>JDBC connect string for a JDBC metastore</description>
        </property>

        <property>
                <name>javax.jdo.option.ConnectionDriverName</name>
                <value>com.mysql.jdbc.Driver</value>
                <description>Driver class name for a JDBC metastore</description>
        </property>

        <property>
                <name>javax.jdo.option.ConnectionUserName</name>
                <value>root</value>
                <description>username to use against metastore database</description>
        </property>

        <property>
                <name>javax.jdo.option.ConnectionPassword</name>
                <value>201012</value>
                <description>password to use against metastore database</description>
        </property>
        <property>
                <name>hive.cli.print.header</name>
                <value>true</value>
        </property>

        <property>
                <name>hive.cli.print.current.db</name>
                <value>true</value>
        </property>

        <property>
                <name>hive.hwi.listen.host</name>
                <value>0.0.0.0</value>
        </property>

        <property>
                <name>hive.hwi.listen.port</name>
                <value>9999</value>
        </property>

        <property>
                <name>hive.hwi.war.file</name>
                <value>lib/hive-hwi-1.2.2.war</value>
        </property>
</configuration>
  • Step 2.4: 编译hive源码,获取HWI以来的web包

第一步: 解压apache-hive-1.2.2-src.tar.gz,具体执行如下命令

[root@BlogMaster ~]# tar -zxvf apache-hive-1.2.2-src.tar.gz 

第二步: 进入该文件解压目录下的hwi,进行如下编译命令

[root@BlogMaster hwi]# jar cvfM0 hive-hwi-1.2.2.war -C web/ .

此时,若出现如下结果,则说明编译成功。

[root@BlogMaster hwi]# jar cvfM0 hive-hwi-1.2.2.war -C web/ .
adding: css/(in = 0) (out= 0)(stored 0%)
adding: css/bootstrap.min.css(in = 90193) (out= 90193)(stored 0%)
adding: img/(in = 0) (out= 0)(stored 0%)
adding: img/glyphicons-halflings-white.png(in = 4352) (out= 4352)(stored 0%)
adding: img/glyphicons-halflings.png(in = 4352) (out= 4352)(stored 0%)
adding: WEB-INF/(in = 0) (out= 0)(stored 0%)
adding: WEB-INF/web.xml(in = 1438) (out= 1438)(stored 0%)
adding: authorize.jsp(in = 2729) (out= 2729)(stored 0%)
adding: diagnostics.jsp(in = 2365) (out= 2365)(stored 0%)
adding: error_page.jsp(in = 1867) (out= 1867)(stored 0%)
adding: index.jsp(in = 1876) (out= 1876)(stored 0%)
adding: left_navigation.jsp(in = 1553) (out= 1553)(stored 0%)
adding: navbar.jsp(in = 1345) (out= 1345)(stored 0%)
adding: session_create.jsp(in = 2690) (out= 2690)(stored 0%)
adding: session_diagnostics.jsp(in = 2489) (out= 2489)(stored 0%)
adding: session_history.jsp(in = 3150) (out= 3150)(stored 0%)
adding: session_kill.jsp(in = 2236) (out= 2236)(stored 0%)
adding: session_list.jsp(in = 2298) (out= 2298)(stored 0%)
adding: session_manage.jsp(in = 6738) (out= 6738)(stored 0%)
adding: session_remove.jsp(in = 2359) (out= 2359)(stored 0%)
adding: session_result.jsp(in = 2488) (out= 2488)(stored 0%)
adding: show_database.jsp(in = 2346) (out= 2346)(stored 0%)
adding: show_databases.jsp(in = 2096) (out= 2096)(stored 0%)
adding: show_table.jsp(in = 4996) (out= 4996)(stored 0%)
adding: view_file.jsp(in = 2653) (out= 2653)(stored 0%)

第三步: 拷贝新生成的war包至hive安装目录下的lib子目录

[root@BlogMaster hwi]# cp hive-hwi-1.2.2.war /opt/cluster/apache-hive-1.2.2-bin/lib/
  • Step 2.5: 配置hive环境变量

进入/etc下的profile文件,新添加对应HIVE_HOME即内部的bin目录,具体如下:

export HIVE_HOME=/opt/cluster/apache-hive-1.2.2-bin
export PATH=$PATH:$JAVA_HOME/bin:$JAVA_HOME/jre:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$HIVE_HOME/bin:$ANT_HOME/bin

之后执行source profile,确保配置生效。

  • Step 2.6: 新增三个jar包至hive安装目录下的lib子目录

为避免出现加载Hive HWI页面时,出现JSP问题,需要将如下三个包拷贝至hive安装目录的lib子目录下。

1. jasper-compiler-5.5.23.jar
2. jasper-runtime-5.5.23.jar
3. commons-el-5.5.23.jar

  • Step 2.7: 新增java的tools.jar包至hive安装目录下的lib子目录
[root@BlogMaster ~]# cp /opt/cluster/jdk1.8.0_181/lib/tools.jar /opt/cluster/apache-hive-1.2.2-bin/lib/
  • Step 2.8: 新增与MySQL连接的jar包至hive安装目录下的lib子目录
[root@BlogMaster ~]# cp mysql-connector-java-5.1.39.jar /opt/cluster/apache-hive-1.2.2-bin/lib
  • Step 2.9: 进入Hive安装目录下conf目录,修改hive-log4j.properties文件

进入该目录后,可能并没有hive-log4j.properties文件,但会有一个相似的hive-log4j.properties.template文件,这里采用cp命令拷贝该文件并重命名为hive-log4j.properties。之后进入hive-log4j.properties,修改其hive.log.dir选项,具体如下:

hive.log.dir=/opt/cluster/apache-hive-1.2.2-bin/logs
  • Step 2.10: 进入Hive安装目录下创建一个名为“logs”的文件夹,用于存放hive运行状态监测的日志记录
[root@BlogMaster apache-hive-1.2.2-bin]# mkdir logs

Step 3: MySQL配置,以免只能hive单窗口操作

MySQL节点可在任意节点安装,但需注意设置其与Hive交互的IP地址的更改。为方便操作,这里MySQL也将部署与BlogMaster节点。 在进行如下操作时,须确保主节点并未配置mysql,若自带,务必直接删除。

判断主节点BlogMaster是否包含MySQL,可执行命令:yum list installed | grep mysql。
若包含,则执行yum -y remove +数据库名称(此处数据库为上述命令执行后出现的MySQL数据库)。

  • Step 3.1: 执行yum install libaio命令,安装依赖环境
[root@BlogMaster ~]# yum install libaio
  • Step 3.2: 执行yum install wget
[root@BlogMaster ~]# yum install wget
  • Step 3.3: 执行wget http://dev.mysql.com/get/mysql-community-release-el7-5.noarch.rpm
[root@BlogMaster ~]# wget http://dev.mysql.com/get/mysql-community-release-el7-5.noarch.rpm
  • Step 3.4: 添加 MySQL Yum Repository 到你的系统 repository 列表中,执行
[root@BlogMaster ~]# yum localinstall mysql-community-release-el7-5.noarch.rpm
  • Step 3.5: 验证下是否添加成功
[root@BlogMaster ~]# yum repolist enabled | grep "mysql.*-community.*"
  • Step 3.6: 验证下是否添加成功
[root@BlogMaster ~]# yum repolist enabled | grep "mysql.*-community.*"

出现如下结果,则说明添加成功。

[root@BlogMaster ~]# yum repolist enabled | grep "mysql.*-community.*"
mysql-connectors-community/x86_64       MySQL Connectors Community           131
mysql-tools-community/x86_64            MySQL Tools Community                100
mysql56-community/x86_64                MySQL 5.6 Community Server           496
  • Step 3.7: 查看 MySQL 版本,执行
[root@BlogMaster ~]# yum repolist all | grep mysql
[root@BlogMaster ~]# yum repolist enabled | grep mysql
  • Step 3.8: 通过 Yum 来安装 MySQL
[root@BlogMaster ~]# yum install mysql-community-server 

此处较长时间的等待,可以听一首徐佳莹的失落沙洲,enjoy…
最后,中间出现提示请选择"y"一路回车。若最终出现如下结果,则说明安装成功,即提示包含了 mysql-community-server、mysql-community-client、mysql-community-common、mysql-community-libs 四个包。

  • Step 3.9: 执行如下命令
[root@BlogMaster ~]# rpm -qi mysql-community-server.x86_64 0:5.6.24-3.el7
  • Step 3.10: 启动MySQL服务
[root@BlogMaster ~]# systemctl start  mysqld
  • Step 3.11: 远程访问 MySQL, 需开放默认端口号 3306.
[root@BlogMaster ~]# firewall-cmd --permanent --zone=public --add-port=3306/tcp
[root@BlogMaster ~]# firewall-cmd --permanent --zone=public --add-port=3306/udp
[root@BlogMaster ~]# firewall-cmd --reload 
  • Step 3.12: MySQL 安全设置,配置MySQLroot用户的密码
[root@BlogMaster ~]# mysql_secure_installation

中间过程中,出现提示直接输入“y”;另外出现设置根用户密码的提示时,设置自己的密码,但请务必与hive-site中设置与MySQL连接的密码一致,即2.3小节的

<property>
		<name>javax.jdo.option.ConnectionURL</name>
		<value>jdbc:mysql://BlogMaster:3306/metastore?createDatabaseIfNotExist=true</value>
		<description>JDBC connect string for a JDBC metastore</description>
</property>

<property>
		<name>javax.jdo.option.ConnectionDriverName</name>
		<value>com.mysql.jdbc.Driver</value>
		<description>Driver class name for a JDBC metastore</description>
</property>

<property>
		<name>javax.jdo.option.ConnectionUserName</name>
		<value>root</value>
		<description>username to use against metastore database</description>
</property>

<property>
		<name>javax.jdo.option.ConnectionPassword</name>
		<value>201012</value>
		<description>password to use against metastore database</description>
</property>
  • Step 3.13: MySQL 安全设置,配置MySQLroot用户的密码

第一步:进入MySQL shell,其中用户为root,密码为自己设置的密码(这里设置的201012)。

[root@BlogMaster ~]# mysql -uroot -p201012
Warning: Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 64
Server version: 5.6.46 MySQL Community Server (GPL)

Copyright (c) 2000, 2019, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

第二步:进入后,选择名称为mysql的数据库,

mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| metastore          |
| mysql              |
| performance_schema |
+--------------------+
4 rows in set (0.11 sec)

mysql> use mysql;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
mysql> 

值得注意,这里出现的metastore即为hive与MySQL交互的hive的元数据。

第三步:查询user表
在这里插入图片描述
第四步:修改 user表,把 Host表内容修改为 %

mysql>update user set host='%' where host='localhost';

第五步:删除root用户的其它host

mysql>delete from user where Host='127.0.0.1';
mysql>delete from user where Host='::1';

第六步:刷新,使设置生效

mysql>flush privileges;

之后执行quit命令,退出MySQL即可。

Part IV - Hive运行状态监测

Step 1: 启动hive之间,务必启动Hadoop集群和Zookeeper服务

在按照此前博客启动Hadoop集群的基础上,执行hive --service hwi命令。

[root@BlogMaster ~]# hive --service hwi

出现如下结果,则说明监测成功。

19/11/13 17:15:59 INFO hwi.HWIServer: HWI is starting up
19/11/13 17:16:10 INFO mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
19/11/13 17:16:10 INFO mortbay.log: jetty-6.1.26
19/11/13 17:16:10 INFO mortbay.log: Extract /opt/cluster/apache-hive-1.2.2-bin/lib/hive-hwi-1.2.2.war to /tmp/Jetty_0_0_0_0_9999_hive.hwi.1.2.2.war__hwi__21w1ka/webapp
19/11/13 17:16:11 INFO mortbay.log: Started SocketConnector@0.0.0.0:9999

Step 2: 执行如下命令查看Hive情况

  • Step 2.1: Jps查看
[root@BlogMaster lib]# jps
7746 NameNode
8291 QuorumPeerMain
15255 Jps
8029 ResourceManager
15182 RunJar

出现RunJar进程,则说明启动成功。

  • Step 2.2:Hive Web页面查看

查看Hive与HDFS交互的数据,登录网址:http://192.168.137.128:50070/explorer.html#/user/hive/warehouse

在这里插入图片描述
直接进入Hive Web页面,查看创建的student_db数据库。登录网址为: http://192.168.137.128:9999/hwi
在这里插入图片描述

  • 0
    点赞
  • 0
    收藏
    觉得还不错? 一键收藏
  • 0
    评论

“相关推荐”对你有帮助么?

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值