大数据集群配置-hive

hadoopHA配置

软件名称	软件包名
hive	apache-hive-2.0.0-src.tar.gz
mysql-connector	mysql-connector-java-5.1.26-bin.jar
CentOS	CentOS 7.4(mini 英文版本)

5.配置主机名和ip

# master 
hostname master && bash
echo master > /etc/hostname
vi /etc/hostname
# slave1 
hostname slave1 && bash
echo slave1 > /etc/hostname
vi /etc/hostname
# slave2
hostname slave2 && bash
echo slave2 > /etc/hostname
vi /etc/hostname

4.配置hosts

# 查看ip
ip a
#master slave1 slave2
vi /etc/hosts
# 例如，ip不代表你本机ip，记得更换
# 三台都需要配置hosts文件

#请根据本机ip ： ip a
192.168.10.3 master
192.168.10.4 slave1
192.168.10.5 slave2

时钟同步

在/etc/sysconfig/ntpd文件添加
YS_HWLOCK=yes
在执行命令
systemctl start ntpd
#查看是否同步时间成功
date

• 如果不行的话，执行下面的命令

#master slave1 slave2 一起执行，三台机子执行间隔越小越好
date -s "20220517 11:31:01"

防火墙关闭

命令如下

#master slalve1 slave2
systemctl stop firewalld.service
systemctl disable firewalld.service

1.JAVA安装

tar -zxvf /opt/software/jdk-8u65-linux-x64.tar.gz -C /usr/local/src
mv /usr/local/src/jdk1.8.0_65 /usr/local/src/java
chown -R root:root /usr/local/src/java
vi /etc/profile

添加如下内容并截图

export JAVA_HOME=/usr/local/src/java
export PATH=$PATH:$JAVA_HOME/bin

保存退出

source /etc/profile
update-alternatives --install /usr/bin/java java /usr/local/src/java/bin/java 200
update-alternatives --set java /usr/local/src/java/bin/java

2.安装ssh

在指定目录下安装ssh服务，查看ssh进程并截图（安装包统一在“/h3cu/”)

rpm -ivh /h3cu/*.rpm

用下面的命令查看ssh服务

rpm -qa | grep ssh

出现以下结果表示安装成功：

[root@master ~]# rpm -qa | grep ssh
openssh-7.4p1-22.el7_9.x86_64
openssh-clients-7.4p1-22.el7_9.x86_64
openssh-server-7.4p1-22.el7_9.x86_64
libssh2-1.8.0-4.el7.x86_64

3.免密配置

下面四条命令需要在三台机器上都要执行，其中ssh-keygen -t rsa 是第一个执行的

ssh-keygen -t rsa 输入后三次回车

[root@master .ssh]$ ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa): 
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:M0qlJuG3H2LvgI4Av1IEitpWKo5I6VGm/mi+Yy97EU8 root@master
The key's randomart image is:
+---[RSA 2048]----+
|                 |
|.                |
|o.  .   .        |
|o .=.E o         |
|oo=o* = S        |
|+*=. B o o       |
|O+o o * .        |
|+X.= . = .       |
|+*@o.  .+        |
+----[SHA256]-----+

ssh-copy-id master

[root@master ~]$ ssh-copy-id master
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
root@master's password: 

Number of key(s) added: 1

Now try logging into the machine, with:   "ssh 'master'"
and check to make sure that only the key(s) you wanted were added.

ssh-copy-id slave1

[root@master ~]$ ssh-copy-id slave1
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
The authenticity of host 'slave1 (192.168.28.112)' can't be established.
ECDSA key fingerprint is SHA256:P0A5pFfF46dyYzSPdTMrfrOxUrLXPK7zbjO0AJggJww.
ECDSA key fingerprint is MD5:e5:fc:9b:bd:90:b6:52:09:0b:91:a8:05:94:b8:6e:c7.
Are you sure you want to continue connecting (yes/no)? yes
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
root@slave1's password: 

Number of key(s) added: 1

Now try logging into the machine, with:   "ssh 'slave1'"
and check to make sure that only the key(s) you wanted were added.

ssh-copy-id slave2

[root@master ~]$ ssh-copy-id slave2
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
The authenticity of host 'slave2 (192.168.28.113)' can't be established.
ECDSA key fingerprint is SHA256:P0A5pFfF46dyYzSPdTMrfrOxUrLXPK7zbjO0AJggJww.
ECDSA key fingerprint is MD5:e5:fc:9b:bd:90:b6:52:09:0b:91:a8:05:94:b8:6e:c7.
Are you sure you want to continue connecting (yes/no)? yes
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
root@slave2's password: 

Number of key(s) added: 1

Now try logging into the machine, with:   "ssh 'slave2'"
and check to make sure that only the key(s) you wanted were added.

截取主节点登录其中一 个从节点的结果

[root@master ~]# ssh slave1
Last login: Sun May 15 11:10:28 2022 from 192.168.10.1
[root@slave1 ~]# 

master分发环境变量文件

#master
scp -r /usr/local/src/java slave1:/usr/local/src
scp -r /usr/local/src/java slave2:/usr/local/src
scp -r /etc/profile slave1:/etc/
scp -r /etc/profile slave2:/etc/

在slave1机器执行如下代码

source /etc/profile
update-alternatives --install /usr/bin/java java /usr/local/src/java/bin/java 200
update-alternatives --set java /usr/local/src/java/bin/java

在slave2机器执行如下代码

source /etc/profile
update-alternatives --install /usr/bin/java java /usr/local/src/java/bin/java 200
update-alternatives --set java /usr/local/src/java/bin/java

Hive

解压hive到指定目录,并为hive添加环境变量

tar -zxvf /opt/software/apache-hive-2.0.0-bin.tar.gz -C /usr/local/src/
mv /usr/local/src/apache-hive-2.0.0-bin/ /usr/local/src/hive
chown -R root:root /usr/local/src/hive
vi /etc/profile

export HIVE_HOME=/usr/local/src/hive
export HIVE_CONF_DIR=$HIVE_HOME/conf
export PATH=$HIVE_HOME/bin:$PATH

格式化环境变量

source /etc/profile

卸载MariaDB

rpm -qa | grep mariadb  #查出MariaDB的文件名称
rpm -qa | grep mariadb | xargs -n1 rpm -e --nodeps

Mysql的安装与启动

移动到Msql安装包所在目录，利用下面命令依次安装common，libs，client，server

cd /opt/software/mysql-5.7.18
rpm -ivh mysql-community-common-5.7.18-1.el7.x86_64.rpm 
rpm -ivh mysql-community-libs-5.7.18-1.el7.x86_64.rpm 
rpm -ivh mysql-community-client-5.7.18-1.el7.x86_64.rpm 
rpm -ivh mysql-community-server-5.7.18-1.el7.x86_64.rpm 

检查安装情况

rpm -qa | grep mysql

修改Mysql的配置文件

vi /etc/my.cnf

下面文件文件，如果没有则全部添加

default-storage-engine=innodb
innodb_file_per_table
collation-server=utf8_general_ci
init-connect='SET NAMES utf8'
character-set-server=utf8

Mysql的服务启动与查看

systemctl start mysqld
systemctl status mysqld

查看mysql的默认密码

cat /var/log/mysqld.log | grep password

重新设定密码和配置，新密码设置为Password123$

mysql_secure_installation

上述命令执行后输入查询到的默认密码，只有Disallow root login remotely?显示的时候设置N其余全部为Y。
N表示允许远程连接

创建hive用户与元数据

mysql -uroot -pPassword123$

create database hive_db;
create user hive identified by 'Password123$';
grant all privileges on *.* to hive@'%' identified by 'Password123$' with grant option;
grant all privileges on *.* to 'root'@'%'identified by 'Password123$' with grant option;
flush privileges;

修改hive的配置文件

#将给的hive-site.xml文件放在Hive的conf目录中
vi /usr/local/src/hive/conf/hive-site.xml
<configuration>
            <property>
                <name>javax.jdo.option.ConnectionURL</name>
                <value>jdbc:mysql://master:3306/hive_db?createDatabaseIfNotExist=true</value>
            </property>
            <!--mysql 用户名-->
            <property>
                <name>javax.jdo.option.ConnectionUserName</name>
                <value>hive</value>
            </property>
            <!--mysql 中 hive 用户密码-->
            <property>
                <name>javax.jdo.option.ConnectionPassword</name>
                <value>Password123$</value>
            </property>
            <!--mysql 驱动-->
            <property>
                <name>javax.jdo.option.ConnectionDriverName</name>
                <value>com.mysql.jdbc.Driver</value>
            </property>
            <property>
                <name>hive.downloaded.resources.dir</name>
                <value>/usr/local/src/hive/tmp</value>
            </property>
            <property>
                <name>hive.exec.local.scratchdir</name>
                <value>/usr/local/src/hive/tmp/${hive.session.id}_resources</value>
            </property>
            <property>
                <name>hive.querylog.location</name>
                <value>/usr/local/src/hive/tmp</value>
            </property>
            <property>
                <name>hive.server2.logging.operation.log.location</name>
                <value>/usr/local/src/hive/tmp/operation_logs</value>
            </property>
            <property>
                <name>hive.server2.webui.host</name>
                <value>master</value>
            </property>
            <property>
                <name>hive.server2.webui.port</name>
                <value>10002</value>
            </property>
</configuration>

修改hive-env.sh,在hive的conf目录下

cd /usr/local/src/hive/conf
mv hive-env.sh.template hive-env.sh
vi hive-env.sh
#修改或添加如下内容
export JAVA_HOME=/usr/local/src/java
export HADOOP_HOME=/usr/local/src/hadoop
export HIVE_CONF_DIR=/usr/local/src/hive/conf
export HIVE_AUX_JARS_PATH=/usr/local/src/hive/lib

将mysql的驱动复制到hive的lib目录下

cp /opt/software/mysql-connector-java-*.jar /usr/local/src/hive/lib/

初始化hive

schematool -initSchema -dbType mysql

启动hive

hive

启动远程hive

修改core-site.xml文件

vi /usr/local/src/hadoop/etc/hadoop/core-site.xml
<property>
	<name>hadoop.proxyuser.root.hosts</name>
	<value>*</value>
</property>
<property>
	<name>hadoop.proxyuser.root.groups</name>
	<value>*</value>
</property>

分发并重启hadoop

#master执行
scp /usr/local/src/hadoop/etc/hadoop/core-site.xml slave1:/usr/local/src/hadoop/etc/hadoop
scp /usr/local/src/hadoop/etc/hadoop/core-site.xml slave2:/usr/local/src/hadoop/etc/hadoop
scp -r /usr/local/src/hive slave1:/usr/local/src
scp /etc/profile slave1:/etc
#重启hadoop
stop-all.sh
start-all.sh

#slave1执行
source /etc/profile

master启动hive远程

#master
hiveserver2

slave1连接

#slave1
beeline -u jdbc:hive2://master:10000