Hive常见问题 unable to create manager for [/var/log/hive/operation_logs/

1. 内存溢出
   虚拟内存溢出:

Current usage: 1.1gb of 2.0gb physical memory used; 4.6gb of 4.2gb virtual memory used. Killing container.==【即虚拟内存溢出】==；

方法一：提高yarn.nodemanager.vmem-pmem-ratio = 5或者更高；【推荐】

方法二：yarn.nodemanager.vmem-check-enabled =false ；关闭虚拟内存检查；不推荐

方法三：提高物理内存分配，相应的虚拟内存自然就多了，但是这样不是最优

物理内存溢出：

Current usage: 2.1gb of 2.0gb physical memory used; 3.6gb of 4.2gb virtual memory used. Killing container.【即物理内存溢出】；

方法一：mapreduce.map.memory.mb = 3GB以上，然后测试这个map/reduce task需要使用多少内存才够用，提高这个值直到不报错为止。

方法二：提高yarn.scheduler.minimum-allocation-mb = 3GB以上，同理【不推荐】

打开低版本hive报错：

ls: cannot access /app/local/spark-2.0.2-bin-hadoop2.6/lib/spark-assembly-*.jar: No such file or directory

修改hive启动文件

vim /app/local/hive/bin/hive

找到下面这一行：

add Spark assembly jar to the classpath

if [[ -n "SPARK_HOME" ]]
then
  # sparkAssemblyPath=`ls{SPARK_HOME}/lib/spark-assembly-*.jar`
  sparkAssemblyPath=`ls {SPARK_HOME}/jars/*.jar`
  CLASSPATH="{CLASSPATH}:${sparkAssemblyPath}"
fi

2. 关联查询

2018-11-25 14:43:04,199 main ERROR Unable to invoke factory method in class class org.apache.hadoop.hive.ql.log element HushableMutableRandomAccess. java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.build(PluginBuilder.java:132)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.createPluginObject(AbstractConfiguration.
    at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration
    at org.apache.logging.log4j.core.appender.routing.RoutingAppender.createAppender(RoutingAppender.java:2
    at org.apache.logging.log4j.core.appender.routing.RoutingAppender.getControl(RoutingAppender.java:255)
    at org.apache.logging.log4j.core.appender.routing.RoutingAppender.append(RoutingAppender.java:225)
    at org.apache.logging.log4j.core.config.AppenderControl.tryCallAppender(AppenderControl.java:156)
    at org.apache.logging.log4j.core.config.AppenderControl.callAppender0(AppenderControl.java:129)
    at org.apache.logging.log4j.core.config.AppenderControl.callAppenderPreventRecursion(AppenderControl.ja
    at org.apache.logging.log4j.core.config.AppenderControl.callAppender(AppenderControl.java:84)
    at org.apache.logging.log4j.core.config.LoggerConfig.callAppenders(LoggerConfig.java:448)
    at org.apache.logging.log4j.core.config.LoggerConfig.processLogEvent(LoggerConfig.java:433)
    at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:417)
    at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:403)
    at org.apache.logging.log4j.core.config.AwaitCompletionReliabilityStrategy.log(AwaitCompletionReliabili
    at org.apache.logging.log4j.core.Logger.logMessage(Logger.java:146)
    at org.apache.logging.log4j.spi.AbstractLogger.logMessageSafely(AbstractLogger.java:2091)
    at org.apache.logging.log4j.spi.AbstractLogger.logMessage(AbstractLogger.java:1993)
    at org.apache.logging.log4j.spi.AbstractLogger.logIfEnabled(AbstractLogger.java:1852)
    at org.apache.logging.slf4j.Log4jLogger.info(Log4jLogger.java:179)
    at org.apache.hadoop.hive.ql.exec.mapjoin.MapJoinMemoryExhaustionHandler.(MapJoinMemoryExhaustion
    at org.apache.hadoop.hive.ql.exec.HashTableSinkOperator.initializeOp(HashTableSinkOperator.java:129)
    at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:358)
    at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:546)
    at org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:498)
    at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:368)
    at org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask.initializeOperators(MapredLocalTask.java:514)
    at org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask.startForward(MapredLocalTask.java:418)
    at org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask.executeInProcess(MapredLocalTask.java:393)
    at org.apache.hadoop.hive.ql.exec.mr.ExecDriver.main(ExecDriver.java:774)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.hadoop.util.RunJar.run(RunJar.java:313)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:227)
Caused by: java.lang.IllegalStateException: ManagerFactory [org.apache.logging.log4j.core.appender.RandomAccessctory@6dac64ea] unable to create manager for [/var/log/hive/operation_logs/5396b439-4945-483d-b8eb-b5c478e6fbb5ae-9f97-23f95080e4be] with data [org.apache.logging.log4j.core.appender.RandomAccessFileManager$FactoryData@5de
    at org.apache.logging.log4j.core.appender.AbstractManager.getManager(AbstractManager.java:114)
    at org.apache.logging.log4j.core.appender.OutputStreamManager.getManager(OutputStreamManager.java:114)
    at org.apache.logging.log4j.core.appender.RandomAccessFileManager.getFileManager(RandomAccessFileManage
    at org.apache.hadoop.hive.ql.log.HushableRandomAccessFileAppender.createAppender(HushableRandomAccessFi
    ... 40 more

异常原因：mr将数据量小的表识别成了大表，数据量大的识别成小表，导致将数据量大的表加入到内存，导致程序异常
处理方法：

set hive.execution.engine=mr;
set hive.mapjoin.smalltable.filesize=55000000;
set hive.auto.convert.join = false;  #取消小表加载至内存中
==通常情况下==，设置取消小表加载至内存中即可：set hive.auto.convert.join = false;

3. hive on spark问题

Job aborted due to stage failure: Aborting TaskSet 2.0 because task 8 (partition 8) cannot run anywhere due to node and executor blacklist. Blacklisting behavior can be configured via spark.blacklist.*.

临时解决办法：

set hive.execution.engine = mr;

https://www.iteblog.com/archives/1907.html

4. hive 事务表
   执行的操作

delete from hm2.history_helper_back where starttime = ‘2019-06-12’;
报错信息

FAILED: SemanticException [Error 10294]: Attempt to do update or delete using transaction manager that does not support these operations.

解决办法：

set hive.support.concurrency = true;
set hive.txn.manager = org.apache.hadoop.hive.ql.lockmgr.DbTxnManager;

5. 修复大量分区
   hive> MSCK REPAIR TABLE employee;
   FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask
   设置：

set hive.msck.path.validation=ignore;

https://stackoverflow.com/questions/40043986/hdinsight-hive-msck-repair-table-table-name-throwing-error

6. hiveserver2 不识别udf函数
   在无法使用UDF的 HiveServer2 上，执行 reload function 命令，将MetaStore中新增的UDF信息同步到HiveServer2内存中。

7. 动态分区

Caused by: org.apache.hadoop.hive.ql.metadata.HiveFatalException: [Error 20004]: Fatal error occurred when node tried to create too many dynamic partitions. The maximum number of dynamic partitions is controlled by hive.exec.max.dynamic.partitions and hive.exec.max.dynamic.partitions.pernode. Maximum was set to 100 partitions per node, number of dynamic partitions on this node: 101
    at org.apache.hadoop.hive.ql.exec.FileSinkOperator.getDynOutPaths(FileSinkOperator.java:941)
    at org.apache.hadoop.hive.ql.exec.FileSinkOperator.process(FileSinkOperator.java:712)
    at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:879)
    at org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:95)
    at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:879)
    at org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:130)
    at org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.forward(MapOperator.java:147)
    at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:487)
    ... 9 more

设置每个节点最大动态分区个数.

set hive.exec.max.dynamic.partitions=3000;
set hive.exec.max.dynamic.partitions.pernode=1000;

SQL
8. block块丢失

Caused by: org.apache.hadoop.hdfs.BlockMissingException: Could not obtain block: BP-808991319-10.1.0.62-1541662386662:blk_1110742285_40900ile=/user/hive/warehouse/hm4.db/hm4_format_log_his_tmp/dt=2019-09-16/hour=11/product=mini/event=click/part-r-00013_copy_1

由于block块受损，无法恢复，只能删除。

作者：z3935212
原文：https://blog.csdn.net/z3935212/article/details/78637157?utm_source=copy