3.12hive 与Hadoop3.2.2 集成/安装遇到的坑

Hadoop3.2.2 版本需要hive3.x
安装mysql 后安装hive

坑1

org.apache.hadoop.conf.Configuration.getTimeDuration

解决办法

hadoop-common.jar 放在hive的lib的文件夹

坑2

The specified datastore driver (“com.mysql.jdbc.Driver”) was not found in the CLASSPATH

解决办法

官网下载mysql-connect-java.jar压缩包，解压后放在hive的lib下

坑3

com.google.common.base.Preconditions.checkArgument

解决办法

hadoop/share/hadoop/common/lib/下的guava jar 替换掉hive/lib下的低版本guava jar

坑4

Required table missing : “DBS” in Catalog “” Schema “”

解决办法

hive的hive-site.xml添加以下配置

<property>
    <name>datanucleus.schema.autoCreateAll</name>
    <value>true</value>
 </property>

坑5

java.lang.NoClassDefFoundError: org/apache/hive/org/apache/commons/cli/Parse

解决办法

看到NoClassDefFoundError，那一定是版本冲突，存在多版本
删除hive/lib下的hive-jdbc-3.1.2-standalone.jar即可

坑6

Caused by: java.lang.ClassNotFoundException: org.apache.tez.dag.api.TezConfiguration

解决办法

hive-site.xml添加以下配置 把引擎换成mr，或者等60s重试后也可正常运行

<property>
   <name>hive.execution.engine</name>
   <value>mr</value>
</property>

hive 乱码

进入mysql 的 metastore库中执行

# 1.修改表字段注解和表注解
alter table COLUMNS_V2 modify column COMMENT varchar(256) character set utf8;
alter table TABLE_PARAMS modify column PARAM_VALUE varchar(4000) character set utf8;
# 2.修改分区字段注解
alter table PARTITION_PARAMS modify column PARAM_VALUE varchar(4000) character set utf8;
alter table PARTITION_KEYS modify column PKEY_COMMENT varchar(4000) character set utf8;
# 3.修改索引注解
alter table INDEX_PARAMS modify column PARAM_VALUE varchar(4000) character set utf8;

# 4.修改hive-site.xml配置文件
<property>
    <name>javax.jdo.option.ConnectionURL</name>
    <value>jdbc:mysql://xxxxx:3306/metastore?createDatabaseIfNotExist=true&amp;useUnicode=true&amp;characterEncoding=UTF-8&amp;useSSL=false</value>
    <description>JDBC connect string for a JDBC metastore</description>
</property>