CDH集群的hive升级

简介

我们的CDH集群使用的是6.3.0版本，默认的hive版本是2.1.1，在使用过程中发现presto创建的orc表，在hive中查询报错，在spark中查询不到数据。但hive或者spark生成的orc表，presto可正常使用。因为需要使用apache atlas，故升级了hive到3.1.2，发现hive中可正确查询presto生成的orc表，但spark依然查询不了，肯定了是hive版本导致的问题，排查以后，替换spark目录下hive目录下的hive的jar包，就可以正确查询presto生成的orc表了。

元数据备份

// An highlighted block
# mysqldump -uroot  -p123456 hive > ./hive.sql

账号密码为hive元数据库的账号密码，库为hive的元数据库

查看hive版本

// An highlighted block
(atlas) [root@mycbigdatan03 mysql]# hive --version
WARNING: Use "yarn jar" to launch YARN applications.
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-6.3.0-1.cdh6.3.0.p0.1279813/lib/hive/lib/log4j-slf4j-impl-2.8.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-6.3.0-1.cdh6.3.0.p0.1279813/jars/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Hive 3.1.2
Git git://HW13934/Users/gates/tmp/hive-branch-3.1/hive -r 8190d2be7b7165effa62bd21b7d60ef81fb0e4af
Compiled by gates on Thu Aug 22 15:01:18 PDT 2019
From source with checksum 0492c08f784b188c349f6afb1d8d9847

停掉hive服务

下载

// An highlighted block
# wget http://mirror.bit.edu.cn/apache/hive/hive-3.1.2/apache-hive-3.1.2-bin.tar.gz  
# tar zxvf apache-hive-3.1.2-bin.tar.gz -C /opt/cloudera/  
# mv -r /opt/cloudera/apache-hive-3.1.2-bin /opt/cloudera/hive

复制hive的jar包

// An highlighted block
//将lib文件夹复制一份lib312文件夹
# cp /opt/cloudera/parcels/CDH/lib/hive/lib /opt/cloudera/parcels/CDH/lib/hive/lib312
//删除hive开头的jar包
# rm -rf /opt/cloudera/parcels/CDH/lib/hive/lib312/hive-*.jar
//将hive开头的jar包替换原来的hive开头的jar包
# cp /opt/cloudera/hive/lib/hive-*.jar /opt/cloudera/parcels/CDH/lib/hive/lib312

修改启动文件

# vim /opt/cloudera/parcels/CDH/lib/hive/bin/hive  
  
# HIVE_LIB=${HIVE_HOME}/lib   //注释本行，改为  
HIVE_LIB=/opt/cloudera/parcels/CDH/lib/hive/lib312

升级元数据库

# cd /opt/cloudera/hive/scripts/metastore/upgrade/mysql
 
mysql> use hive;  
Reading table information for completion of table and column names  
You can turn off this feature to get a quicker startup with -A  
  
Database changed  
mysql> source upgrade-2.3.7-to-3.0.0.mysql.sql;  
+------------------------------------------------+  
| |  
+------------------------------------------------+  
| Upgrading MetaStore schema from 2.3.7-to-3.0.0 |  
+------------------------------------------------+  
1 row in set, 1 warning (0.00 sec)  
  
Query OK, 0 rows affected (0.05 sec)  
Records: 0 Duplicates: 0 Warnings: 0  
  
Query OK, 30 rows affected (0.01 sec)  
Rows matched: 30 Changed: 30 Warnings: 0  
  
Query OK, 0 rows affected (0.06 sec)  
Records: 0 Duplicates: 0 Warnings: 0  
  
Query OK, 0 rows affected (0.02 sec)  
Records: 0 Duplicates: 0 Warnings: 0  
  
Query OK, 0 rows affected (0.04 sec)  
Records: 0 Duplicates: 0 Warnings: 0  
  
Query OK, 0 rows affected (0.00 sec)  
Records: 0 Duplicates: 0 Warnings: 0  
  
Query OK, 191 rows affected (0.04 sec)  
Records: 191 Duplicates: 0 Warnings: 0  
  
Query OK, 77 rows affected (0.04 sec)  
Records: 77 Duplicates: 0 Warnings: 0  
  
Query OK, 0 rows affected (0.03 sec)  
Records: 0 Duplicates: 0 Warnings: 0  
  
Query OK, 0 rows affected (0.01 sec)  
Records: 0 Duplicates: 0 Warnings: 0  
  
Query OK, 0 rows affected (0.03 sec)  
Records: 0 Duplicates: 0 Warnings: 0  
  
Query OK, 0 rows affected (0.01 sec)  
Records: 0 Duplicates: 0 Warnings: 0  
  
Query OK, 0 rows affected (0.00 sec)  
Records: 0 Duplicates: 0 Warnings: 0  
  
Query OK, 0 rows affected (0.00 sec)  
Records: 0 Duplicates: 0 Warnings: 0  
  
Query OK, 0 rows affected (0.03 sec)  
Records: 0 Duplicates: 0 Warnings: 0  
  
Query OK, 0 rows affected (0.01 sec)  
Records: 0 Duplicates: 0 Warnings: 0  
  
Query OK, 0 rows affected (0.00 sec)  
Records: 0 Duplicates: 0 Warnings: 0  
  
Query OK, 0 rows affected (0.00 sec)  
Records: 0 Duplicates: 0 Warnings: 0  
  
Query OK, 0 rows affected (0.01 sec)  
Records: 0 Duplicates: 0 Warnings: 0  
  
Query OK, 0 rows affected (0.00 sec)  
Records: 0 Duplicates: 0 Warnings: 0  
  
Query OK, 0 rows affected (0.00 sec)  
Records: 0 Duplicates: 0 Warnings: 0  
  
Query OK, 1 row affected (0.00 sec)  
Rows matched: 1 Changed: 1 Warnings: 0  
  
+---------------------------------------------------------+  
| |  
+---------------------------------------------------------+  
| Finished upgrading MetaStore schema from 2.3.7-to-3.0.0 |  
+---------------------------------------------------------+  
1 row in set, 1 warning (0.00 sec)  
  
mysql> source upgrade-3.0.0-to-3.1.0.mysql.sql;  
+------------------------------------------------+  
| |  
+------------------------------------------------+  
| Upgrading MetaStore schema from 3.0.0-to-3.1.0 |  
+------------------------------------------------+  
1 row in set, 1 warning (0.00 sec)  
  
Query OK, 0 rows affected (0.01 sec)  
Records: 0 Duplicates: 0 Warnings: 0  
  
Query OK, 1 row affected (0.01 sec)  
Rows matched: 1 Changed: 1 Warnings: 0  
  
+---------------------------------------------------------+  
| |  
+---------------------------------------------------------+  
| Finished upgrading MetaStore schema from 3.0.0 to 3.1.0 |  
+---------------------------------------------------------+  
1 row in set, 1 warning (0.00 sec)

注意：不要跳着升级，一步一步来。

同步元数据

我这里操作没成功，但不影响。

替换spark的jar包

// An highlighted block
# spark目录
# cd /opt/cloudera/parcels/CDH/lib/spark
#备份
cp -r hive hive-2.2.1

# 删除hive下jar包
rm -rf hive/*

#拷贝jar包
cp -r /opt/cloudera/parcels/CDH/lib/hive/lib312/* hive/

后记

根据这次升级推测，所有底层用到hive的服务，如果需要升级，都需要更新jar包。
这篇文章是在别人基础上做了一下补充，如有侵权，请联系我删除。

参考文献：
HIVE2升级到HIVE3.1.2: 升级.