使用hive3.x 问题记录

使用 hive3.1.2 问题记录

搭建步骤省略 ...

一、HIVE ON MR

1、第二次以上插入值 insert into table tablename values("xx")； 出现错误:

1)、metastore.RetryingMetaStoreClient: MetaStoreClient lost connection. Attempting to reconnect (1 of 1) after 1s. setPartitionColumnStatistics
org.apache.thrift.transport.TTransportException: null

解决：当hiveserver2和 metastore 服务在一个机器会出现这种错误 分两台机器起就会报2错误 这才是本质

2)、 Error occurred during processing of message.
java.lang.ClassCastException: org.apache.hadoop.hive.metastore.api.StringColumnStatsData cannot be cast to org.apache.hadoop.hive.metastore.columnstats.cache.StringColumnStatsDataInspector

解释一：网上搜索是一种Bug
https://issues.apache.org/jira/browse/HIVE-20151
https://issues.apache.org/jira/browse/HIVE-19316
解释二:
列状态为True会触发这种问题

临时解决:
1、建表之前设置set hive.stats.autogather=false；
2、或者建表之后 执行load data inpath “/xxx.txt” into table tal_name;
1、2 方法 可以去掉表的列状态为True的情况 即可重复执行INSERT INTO tb_name values

一、HIVE ON TEZ

说明：使用的是hive3.1.2 + tez0.9.2

1、 错误信息
java.lang.NoSuchMethodError:
com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;Ljava/lang/Object;)V

网上有人说将hadoop /share/hadoop/common/lib/guava-27.0-jre.jar
路径下的jar替换 tez/lib下的低版本guava jar就可以
我试了还是报这种错误
于是我将tez升级到0.10版本解决这个问题

2、执行卡住 是申请的资源不足导致

通过设置set hive.tez.container.size=512解决

<property>
<name>hive.tez.container.size</name>
<value>-1</value>
<description>By default Tez will spawn containers of the size of a mapper. This can be used to overwrite.</description>   </property>

其他影响参数可以尝试设置解决 set hive.tez.java.opts=-Xmx1640m; set
tez.runtime.io.sort.mb=820; set
tez.runtime.unordered.output.buffer.size-mb=205;

其余相关配置参考如下：
yarn-site配置

   <property>
        <name>yarn.nodemanager.env-whitelist</name>
        <value>JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOME</value>
    </property>

    <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>

    <property>
        <name>yarn.resourcemanager.hostname</name>
        <value>hadoop1</value>
    </property>
    <!-- 是否开启聚合日志 -->
    <property>
        <name>yarn.log-aggregation-enable</name>
        <value>true</value>
    </property>
      <property>
        <name>yarn.scheduler.minimum-allocation-mb</name>
        <value>512</value>
    </property>
    <property>
        <name>yarn.scheduler.maximum-allocation-mb</name>
        <value>8192</value>
    </property>
    <property>
        <name>yarn.nodemanager.resource.memory-mb</name>
        <value>4096</value>
    </property>
    <property>
        <name>yarn.nodemanager.pmem-check-enabled</name>
        <value>false</value>
    </property>

   <property>
        <name>yarn.nodemanager.vmem-check-enabled</name>
        <value>false</value>
    </property>
   <property>
        <name>yarn.nodemanager.resource.cpu-vcores</name>
        <value>8</value>
    </property>


    <!-- 聚合日志在hdfs上的url -->
    <property>
        <name>yarn.log.server.url</name>
        <value>http://hadoop1:19888/jobhistory/logs</value>
    </property>

    <!-- 参数说明：聚合后的日志在HDFS上保存多长时间，单位为s。-->
    <property>
        <name>yarn.log-aggregation.retain-seconds</name>
        <value>604800</value>
    </property>

    <property>
       <name>yarn.resourcemanager.webapp.address.rm1</name>
       <value>hadoop0:8088</value>
    </property>

    <property>
       <name>yarn.resourcemanager.webapp.address.rm2</name>
       <value>hadoop1:8088</value>
    </property>

    <!--启用resourcemanager ha-->
    <property>
        <name>yarn.resourcemanager.ha.enabled</name>
        <value>true</value>
    </property>

    <!--声明两台resourcemanager的地址-->
    <property>
        <name>yarn.resourcemanager.cluster-id</name>
        <value>cluster-yarn1</value>
    </property>

    <property>
        <name>yarn.resourcemanager.ha.rm-ids</name>
        <value>rm1,rm2</value>
    </property>

    <property>
        <name>yarn.resourcemanager.hostname.rm1</name>
        <value>hadoop0</value>
    </property>

    <property>
        <name>yarn.resourcemanager.hostname.rm2</name>
        <value>hadoop1</value>
    </property>

    <!--指定zookeeper集群的地址-->
    <property>
        <name>yarn.resourcemanager.zk-address</name>
        <value>hadoop0:2181,hadoop1:2181,hadoop2:2181</value>
    </property>

    <!--启用自动恢复-->
    <property>
        <name>yarn.resourcemanager.recovery.enabled</name>
        <value>true</value>
    </property>

    <!--指定resourcemanager的状态信息存储在zookeeper集群-->
    <property>
        <name>yarn.resourcemanager.store.class</name>
        <value>org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore</value>
    </property>
</configuration>

tez-site配置

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
        <name>tez.lib.uris</name>
    <value>${fs.defaultFS}/tez/tez.tar.gz</value>
</property>
<property>
     <name>tez.use.cluster.hadoop-libs</name>
     <value>false</value>
</property>
<property>
     <name>tez.history.logging.service.class</name>
     <value>org.apache.tez.dag.history.logging.ats.ATSHistoryLoggingService</value>
</property>

<property>
     <name>tez.am.resource.memory.mb</name>
     <value>1024</value>
</property>

<property>
     <name>tez.am.resource.cpu.vcores</name>
     <value>1</value>
</property>
<property>
     <name>tez.task.resource.memory.mb</name>
     <value>1024</value>
</property>
</configuration>

hive-site配置

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
   <property>
        <name>hive.metastore.uris</name>
        <value>thrift://192.168.187.192:9083</value>
    </property>
   <property>
       <name>hive.execution.engine</name>
      <value>tez</value>
  </property>
  <property>
      <name>hive.tez.container.size</name>
      <value>1024</value>  
   </property>
</configuration>