Hive安装到配置
前言
hadoop version :hadoop 2.7.3
mysql version :MySQL 5.7.32
hive verson :Hive 2.2.0
mysql–connector-*.jar version :mysql-connecter - * .jar 5.1.43
(一)Hive集群规划
角色 | Master | test2 | test3 |
---|---|---|---|
MySql | ● | × | × |
Metstore | × | ● | × |
CLI | × | × | ● |
(二)安装MySql
wget \
https://cdn.mysql.com/archives/mysql-5.7/mysql-community-client-5.7.32-1.el7.x86_64.rpm \
https://cdn.mysql.com/archives/mysql-5.7/mysql-community-common-5.7.32-1.el7.x86_64.rpm \
https://cdn.mysql.com/archives/mysql-5.7/mysql-community-libs-5.7.32-1.el7.x86_64.rpm \
https://cdn.mysql.com/archives/mysql-5.7/mysql-community-libs-compat-5.7.32-1.el7.x86_64.rpm \
https://cdn.mysql.com/archives/mysql-5.7/mysql-community-server-5.7.32-1.el7.x86_64.rpm
yum install -y mysql-community-*-5.7.32-1.el7.x86_64.rpm
# 开启MySQL服务器
systemctl start mysqld
# 查看默认生成的密码
cat /var/log/mysqld.log | grep password
#使用你查到的临时密码进行登录
mysql -u root -p
#设置密码登记
set global validate_password_length=4;
set global validate_password_policy=0;
#修改默认密码
ALTER USER 'root'@'localhost' IDENTIFIED WITH mysql_native_password BY '密码'; #此处注意修改您的密码
-
默认表的字符集是拉丁文,后续录入中文会有乱码,所以需要将如下两个改为
UTF-8
character_set_databse
character_set_server
-
修改配置文件
vim /etc/my.cnf
-
修改后的内容
# For advice on how to change settings please see # http://dev.mysql.com/doc/refman/5.7/en/server-configuration-defaults.html [client] default-character-set=utf8 [mysql] default-character-set=utf8 [mysqld] character-set-server=utf8 # # Remove leading # and set to the amount of RAM for the most important data # cache in MySQL. Start at 70% of total RAM for dedicated server, else 10%. # innodb_buffer_pool_size = 128M # # Remove leading # to turn on a very important data integrity option: logging # changes to the binary log between backups. # log_bin # # Remove leading # to set options mainly useful for reporting servers. # The server defaults are faster for transactions and fast SELECTs. # Adjust sizes as needed, experiment to find the optimal values. # join_buffer_size = 128M # sort_buffer_size = 2M # read_rnd_buffer_size = 2M datadir=/var/lib/mysql socket=/var/lib/mysql/mysql.sock # Disabling symbolic-links is recommended to prevent assorted security risks symbolic-links=0 log-error=/var/log/mysqld.log pid-file=/var/run/mysqld/mysqld.pid
-
重点部分
[client] default-character-set=utf8 [mysql] default-character-set=utf8 [mysqld] character-set-server=utf8
修改后重启MySql服务
(三)Hive安装
-
下载后上传到Centos7对应文件夹再进行解压
# 使用root用户的话,默认目录是在root # 上传的位置是 /root/ # 解压缩文件 tar -zxf apache-hive-2.2.0-bin.tar.gz # 重命名文件夹并将文件夹移动到/usr/local目录下 mv apache-hive-2.2.0-bin.tar.gz /usr/local/Hive
-
配置Hive
-
环境变量配置
vi + /etc/profile
在环境变量中添加如下:
# Hive环境变量 export HIVE_HOME=/usr/local/hive #这里就是你存放Hive的路径 export PATH=$PATH:${HIVE_HOME}/bin
保存更新后需要更新环境变量
source /etc/profile
-
第一个配置文件
Hive-site.xml
# 将Hive文件夹里面的Hive-default.xml.template改名为Hive-site.xml mv ./Hive-default.xml.template ./Hive-site.xml # 修改Hive-site.xml配置文件 vi + /Hive-site.xml
- Hive-site.xml文件的修改内容
<?xml version="1.0" encoding="UTF-8" standalone="no"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> <configuration> <!--mysql 数据库连接的IP地址需要更改为你mysql那台机器的地址--> <property> <name>javax.jdo.option.ConnectionURL</name> <value>jdbc:mysql://192.168.131.132:3306/hive</value> </property> <!--mysql驱动--> <property> <name>javax.jdo.option.ConnectionDriverName</name> <value>com.mysql.jdbc.Driver</value> </property> <!--mysql用户名--> <property> <name>javax.jdo.option.ConnectionUserName</name> <value>root</value> </property> <!--mysql中hive用户密码--> <property> <name>javax.jdo.option.ConnectionPassword</name> <value>123456</value> </property> <property> <name>hive.metastore.schema.verification</name> <value>false</value> </property> </configuration>
-
第二个配置文件
hive-env.sh
# 这是我的环境配置(供参考) # 所有的软件路径根据主机的安装路径来决定 export JAVA_HOME=/usr/local/java export HADOOP_HOME=/usr/local/hadoop export HIVE_HOME=/usr/local/hive
-
初始化Hive(注意:初始化之前需要打开Hadoop)
schematool -dbType mysql -initSchema
如果初始化不成功,详看问题解决【问题解决】
-
验证是否安装成功(能够进入到这个界面表示单机已经安装完成了):
hive
-
-
配置其他节点的Hive
- 将安装包传到其他节点
scp -r 文件路径 用户@主机名:存放路径 # 示例 : scp -r /usr/local/hive root@test1:/usr/local
- 其他节点主机的配置(将其它两个连接到Master主机)
[hadoop@hadoop02 ~]$ vim hive/conf/hive-site.xml
<configuration>
<property>
<name>hive.metastore.uris</name>
<value>thrift://Master:9083</value>
</property>
</configuration>
[hadoop@hadoop03 ~]$ vim hive/conf/hive-site.xml
<configuration>
<property>
<name>hive.metastore.uris</name>
<value>thrift://Master:9083</value>
</property>
</configuration>
(四)问题解决(问题还不全面,有疑问的可以提出一起解决)
-
问题【Hadoop配置安装】
Hadoop安装后jps查询,会有如下几个进程:
如果没有这几个进程,建议直接跳转到:【Hadoop安装配置】
-
问题(1):无法完成初始化
Error: Duplicate key name 'PCS_STATS_IDX' (state=42000,code=1061)org.apache.hadoop.hive.metastore.HiveMetaException: Schema initialization FAILED! Metastore state would be inconsistent !!
原因:MySQL中已经存在Hive需要的表
解决方式:删除MySQL中的所有表然后再进行初始化
-
问题(2):配置文件问题
报错现象(如果是按照上面的
hive-env.sh
进行配置的话不会出现这个问题):[root@master bin]# ./schematool -dbType mysql -initSchema Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512M; support was removed in 8.0 which: no hbase in (/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/usr/java/jdk1.8.0_191/bin:/root/bin) Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512M; support was removed in 8.0 Metastore connection URL: jdbc:mysql://192.168.52.26:3306/hive Metastore Connection Driver : com.mysql.jdbc.Driver Metastore connection User: APP org.apache.hadoop.hive.metastore.HiveMetaException: Failed to get schema version. *** schemaTool failed ***
解决方法:
hive-site.xml
文件中添加以下内容<property> <name>javax.jdo.option.ConnectionURL</name> <value>jdbc:mysql://192.168.52.26:3306/hive</value> </property> <property> <name>javax.jdo.option.ConnectionDriverName</name> <value>com.mysql.jdbc.Driver</value> </property> <property> <name>javax.jdo.option.ConnectionUserName</name> <value>root</value> </property> <property> <name>javax.jdo.option.ConnectionPassword</name> <value>root</value> </property>
-
问题(3):mysql-connecter 驱动问题 :不同的版本对应的不同的驱动,驱动也很有可能导致无法初始化