最详细的Hadoop+Hbase+Hive完全分布式环境搭建教程（二）

最新推荐文章于 2022-01-26 17:45:23 发布

想做一只开心的菜鸡

最新推荐文章于 2022-01-26 17:45:23 发布

阅读量6.6k

点赞数 10

分类专栏：大杂烩文章标签： hive hbase

本文链接：https://blog.csdn.net/Fei20140908/article/details/84036524

版权

大杂烩专栏收录该内容

27 篇文章 0 订阅

订阅专栏

继续上次的教程，这次安装hbase与hive。之前有同学反应百度网盘中没有hbase安装包，这次补上了，在上篇博客的网盘链接中已经有了，最详细的Hadoop+Hbase+Hive完全分布式环境搭建教程（一）。

一、hbase2.1.1安装

在master节点：

$ tar -zxvf hbase-2.1.1-bin.tar.gz -C /opt
$ cd /opt/hbase-2.1.1/conf/
$ vi hbase-env.sh

将下面这句话加到文件末尾
export JAVA_HOME=/opt/jdk1.8.0_102

$ vi hbase-site.xml

修改如下：

<configuration>
<property>
  <name>hbase.rootdir</name>
  <value>hdfs://master:9000/hbase</value>
</property>
<property>
  <name>hbase.zookeeper.quorum</name>
  <value>master,slave1,slave2</value>
</property>
<property>
  <name>hbase.cluster.distributed</name>
  <value>true</value>
</property>
<property>
  <name>hbase.zookeeper.property.dataDir</name>
  <value>/home/fay/zookeeper</value>
</property>
</configuration>

$ vi regionservers
修改如下：

# 删掉localhost
slave1
slave2

$ vi backup-masters
这个文件是新建的，加入下面这句话

slave1

然后就是配置环境变量：
$ vi ~/.bashrc

export JAVA_HOME=/opt/jdk1.8.0_102
export HADOOP_HOME=/opt/hadoop-2.8.5
export HBASE_HOME=/opt/hbase-2.1.1
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib/native"
export PATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$HBASE_HOME/bin:

$ source ~/.bashrc
然后将配置文件和整个目录给其它两个节点：

$ scp -r hbase-2.1.1/ fay@slave1:/opt
$ scp -r hbase-2.1.1/ fay@slave2:/opt
$ scp ~/.bashrc fay@slave1:/home/fay
$ scp ~/.bashrc fay@slave1:/home/fay

另外两个节点source一下
$ source ~/.bashrc
然后在hadoop起来的情况下，再启动hbase

# 启动hadoop
$ start-dfs.sh
$ start-yarn.sh
# 启动hbase
$ start-hbase.sh

然后master节点：

[fay@master opt]$ jps
39825 Jps
38898 ResourceManager
39398 HQuorumPeer #hbase的zookeeper实例
38487 NameNode
39495 HMaster # hbase master
38702 SecondaryNameNode

slave节点：

[fay@slave1 ~]$ jps
16481 DataNode
16598 NodeManager
16886 HRegionServer # hbase regionserver
16775 HQuorumPeer 
17003 HMaster # 这个是slave1，因为前面配置了backup-masters,slave2没有这个
17278 Jps

这里注意时间同步问题，启动可能比较耗时，不要轻易ctrl+c取消，可能会异常。好了，测试一下

[fay@master opt]$ hbase shell
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/hadoop-2.8.5/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/hbase-2.1.1/lib/client-facing-thirdparty/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
HBase Shell
Use "help" to get list of supported commands.
Use "exit" to quit this interactive shell.
For Reference, please visit: http://hbase.apache.org/2.0/book.html#shell
Version 2.1.1, rb60a92d6864ef27295027f5961cb46f9162d7637, Fri Oct 26 19:27:03 PDT 2018
Took 0.0047 seconds                                                                                                                                               
hbase(main):001:0> list
TABLE                                                                                                                                                             
wuxia                                                                                                                                                             
1 row(s) # 这里我之前创建了表，初始状态应该是0
Took 1.8662 seconds                                                                                                                                               
=> ["wuxia"]
hbase(main):002:0> create 'hello','world'
Created table hello
Took 3.1974 seconds                                                                                                                                               
=> Hbase::Table - hello

不出异常基本上hbase 搭好了

二、hive2.3.4搭建

1. 安装mysql 8.0.13 (不安装可以使用自带的derby)

卸载系统自带的mariadb-lib
查看mariadb版本
rpm -qa|grep mariadb

卸载mariadb
rpm -e mariadb-libs-5.5.56-2.el7.x86_64 --nodeps
mysql有依赖关系，安装顺序是 common、lib、client、server

sudo  rpm -ivh mysql-community-common-8.0.13-1.el7.x86_64.rpm
sudo rpm -ivh mysql-community-libs-8.0.13-1.el7.x86_64.rpm
sudo rpm -ivh mysql-community-client-8.0.13-1.el7.x86_64.rpm
sudo rpm -ivh mysql-community-server-8.0.13-1.el7.x86_64.rpm

启动mysql：

sudo systemctl start mysqld
设置root密码：

 [fay@master ~]$ cat /var/log/mysqld.log
2018-11-11T09:12:06.253251Z 0 [System] [MY-013169] [Server] /usr/sbin/mysqld (mysqld 8.0.13) initializing of server in progress as process 20113
2018-11-11T09:12:09.192846Z 5 [Note] [MY-010454] [Server] A temporary password is generated for root@localhost: pR+ndHtvO6#d

上面的pR+ndHtvO6#d就是root初始密码，用这个密码登录mysql

[fay@master ~]$ mysql -u root -p
Enter password: 
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 19
Server version: 8.0.13 MySQL Community Server - GPL

Copyright (c) 2000, 2018, Oracle and/or its affiliates. All rights reserved.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql>

然后就是创建一个fay用户，并给予他权限

mysql> CREATE USER 'fay'@'master' IDENTIFIED BY '你的密码';
mysql> create database hive_metedata
mysql> grant all privileges on *.* to 'fay'@'master';
mysql> flush privileges;

2. 安装hive

解压到/opt文件夹：
$ tar -zxvf apache-hive-2.3.4-bin.tar.gz -C /opt
$ cd /opt/apache-hive-2.3.4-bin/conf/
修改 hive-env.sh
配置hadoop_home等环境变量，在文件末尾有

[fay@master conf]$ cp hive-env.sh.template hive-env.sh
[fay@master conf]$ vi hive-env.sh
...
# HADOOP_HOME=${bin}/../../hadoop
export HADOOP_HOME=/opt/hadoop-2.8.5

# Hive Configuration Directory can be controlled by:
export HIVE_CONF_DIR=/opt/apache-hive-2.3.4-bin/conf

# Folder containing extra libraries required for hive compilation/execution can be controlled by:
export HIVE_AUX_JARS_PATH=/opt/apache-hive-2.3.4-bin/lib
export JAVA_HOME=/opt/jdk1.8.0_102

修改hive-site.xml

[fay@master conf]$ cp hive-default.xml.template hive-site.xml 
[fay@master conf]$ vi hive-site.xml 
# 文件太长了，就copy几个重要需要改的地方，找到这些配置项改
<property>
    <name>hive.exec.local.scratchdir</name>
    <value>/home/fay/hive/tmp/fay</value>
    <description>Local scratch space for Hive jobs</description>
  </property>
<property>
    <name>hive.downloaded.resources.dir</name>
    <value>/home/fay/hive/tmp/${hive.session.id}_resources</value>
    <description>Temporary local directory for added resources in the remote file system.</description>
</property>
 <property>
    <name>javax.jdo.option.ConnectionPassword</name>
    <value>你的密码</value>
    <description>password to use against metastore database</description>
  </property>
  <property>
    <name>javax.jdo.option.ConnectionURL</name>
    <value>jdbc:mysql://master:3306/hive_metedata?createDatabaseIfNotExist=true</value>
    <description>
      JDBC connect string for a JDBC metastore.
      To use SSL to encrypt/authenticate the connection, provide database-specific SSL flag in the connection URL.
      For example, jdbc:postgresql://myhost/db?ssl=true for postgres database.
    </description>
  </property>
<property>
    <name>javax.jdo.option.ConnectionDriverName</name>
    <value>com.mysql.cj.jdbc.Driver</value>    
    <description>Driver class name for a JDBC metastore</description>
  </property>

注：javax.jdo.option.ConnectionDriverName如果是com.mysql.jdbc.Driver，就要把那个mysql-connector jar包放入到${hive_home}/lib文件夹里,所以那个mysql-connector jar包可有可无

然后将配置文件中所有的${system:java.io.tmpdir}更改为 /home/fay/hive/tmp (如果没有该文件则创建)，并将此文件夹赋予读写权限，将${system:user.name}更改为 fay

然后添加环境变量：

[fay@master lib]$ vi ~/.bashrc

# .bashrc

# Source global definitions
if [ -f /etc/bashrc ]; then
        . /etc/bashrc
fi

# Uncomment the following line if you don't like systemctl's auto-paging feature:
# export SYSTEMD_PAGER=

# User specific aliases and functions

export JAVA_HOME=/opt/jdk1.8.0_102
export HADOOP_HOME=/opt/hadoop-2.8.5
export HBASE_HOME=/opt/hbase-2.1.1
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib/native"
export HIVE_HOME=/opt/apache-hive-2.3.4-bin
export PATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$HBASE_HOME/bin:$HIVE_HOME/bin

source ~/.bashrc
创建hdfs文件夹，并赋予权限

$ hadoop fs -mkdir -p /user/hive/
$ hadoop fs -mkdir -p /user/hive/warehouse
$ hadoop fs -chmod 777 /user/hive/
$ hadoop fs -chmod 777 /user/hive/warehouse

初始化hive，然后启动hive：

[fay@master conf]$ schematool -dbType mysql -initSchema
[fay@master conf]$ hive
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/apache-hive-2.3.4-bin/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/hadoop-2.8.5/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]

Logging initialized using configuration in jar:file:/opt/apache-hive-2.3.4-bin/lib/hive-common-2.3.4.jar!/hive-log4j2.properties Async: true
Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
hive>

然后测试下hive的功能，与sql差不多，这里不过多介绍，到这里，全部环境搭建完啦。如果中途遇到问题，网上还是有不少解决方案的，后面如果我按这个再搭一次环境出问题再来调整。环境多搭建就熟悉了，我也是搭建过将近10次的环境，以前都是用一个版本，这次打算用最新的版本，中间还是有不少坑的。