一, 初始化元数据出错
使用derby作为元数据库,初始化时失败。
/export/server/hive-2.1.0/bin/schematool -dbType derby -initSchema
0: jdbc:derby:> CREATE FUNCTION "APP"."NUCLEUS_ASCII" (C CHAR(1)) RETURNS INTEGER LANGUAGE JAVA PARAMETER STYLE JAVA READS SQL DATA CALLED ON NULL INPUT EXTERNAL NAME 'org.datanucleus.store.rdbms.adapter.DerbySQLFunction.ascii'
Error: FUNCTION 'NUCLEUS_ASCII' already exists. (state=X0Y68,code=30000)
Closing: 0: jdbc:derby:;databaseName=metastore_db;create=true
org.apache.hadoop.hive.metastore.HiveMetaException: Schema initialization FAILED! Metastore state would be inconsistent !!
Underlying cause: java.io.IOException : Schema script failed, errorcode 2
org.apache.hadoop.hive.metastore.HiveMetaException: Schema initialization FAILED! Metastore state would be inconsistent !!
at org.apache.hive.beeline.HiveSchemaTool.doInit(HiveSchemaTool.java:291)
at org.apache.hive.beeline.HiveSchemaTool.doInit(HiveSchemaTool.java:264)
at org.apache.hive.beeline.HiveSchemaTool.main(HiveSchemaTool.java:505)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by: java.io.IOException: Schema script failed, errorcode 2
at org.apache.hive.beeline.HiveSchemaTool.runBeeLine(HiveSchemaTool.java:390)
at org.apache.hive.beeline.HiveSchemaTool.runBeeLine(HiveSchemaTool.java:347)
at org.apache.hive.beeline.HiveSchemaTool.doInit(HiveSchemaTool.java:287)
原因:已经执行过mysql的初始化,使用derby导致冲突,将以前的数据库改名可以解决这个问题。
cd /export/server/hive-2.1.0/scripts/metastore/upgrade/derby
mv metastore_db metastore_db.tmp
然后重新初始化,即可成功。
二,FAILED: SemanticException Line 0:-1 Partition not found
这种情况一般是你创建的表是多级分区(比如year,month),若操作时指定一个分区是不可以的(只加day或者hour)
insert overwrite table test partition (
year=2021
month='17'
)
三,ob Submission failed with exception 'java.io.IOException(org.apache.hadoop.yarn.exceptions.YarnException: Failed to submit application_1621563516281_0049 to YARN : Application application_1621563516281_0049 submitted by user root to unknown queue: ‘prod’
设置队列名称时不要用引号:
#正确
set mapreduce.job.queuename=prod;
#错误
set mapreduce.job.queuename='prod';
四,Could not connect to hadoop02:10000 (code THRIFTTRANSPORT): TTransportException(‘Could not connect to hadoop02:10000’,)
由于 HiveServer2 的 Java 堆栈大小(字节)过小,导致意外退出
五,Hive执行sql语句,提交了之后一直卡住
查看Yarn,发现是UNDEFINED状态:
再点进去查看日志,发现资源不足:
Resource request: <memory:1024, vCores:1> exceeds maximum AM resource allowed
查看NodeManager的配置,发现 yarn.nodemanager.resource.memory-mb=1G
,nodemanager只配了1G的内存,表示该节点上YARN可使用的物理内存总量,而Map任务需要1G的内存,自然不够了。
解决办法就是扩大这个配置的值。
但这个值和yarn.scheduler.minimum-allocation-mb
以及yarn.scheduler.maximum-allocation-mb
有关系,必须要介于二者直接。
这两个值在yarn的web ui上可以看到:
六,MapReduce任务卡住不动
ACCEPTED: waiting for AM container to be allocated, launched and register with RM.
应该也是内存问题,见上一个问题的第一种解法。