一日一坑之：hadoop3.x VS hive3.x

最新推荐文章于 2024-05-03 11:06:55 发布

wtysuper6

最新推荐文章于 2024-05-03 11:06:55 发布

阅读量3.1k

点赞数 1

分类专栏： hive 文章标签： hive错误 hive坑 hive3问题 hadoop3与hive3 hive坑解决办法

本文链接：https://blog.csdn.net/a763470525/article/details/100563978

版权

hive 专栏收录该内容

2 篇文章 0 订阅

订阅专栏

最近安装了新版本的hadoop3.x版本，按照老思路此时需要安装hive3.x用来搭配我们崭新的hadoop，重新翻出来我的泛黄的hive2.x版本的安装说明书安装了一遍，就在此时坑来了。

坑一：FAILED: ParseException line 1:5 cannot recognize input near 'show' 'database' '<EOF>' in ddl statement

错误信息：

NoViableAltException(70@[651:1: ddlStatement : ( createDatabaseStatement | switchDatabaseStatement | dropDatabaseStatement | createTableStatement | dropTableStatement | truncateTableStatement | alterStatement | descStatement | showStatement | metastoreCheck | createViewStatement | dropViewStatement | createFunctionStatement | createMacroStatement | createIndexStatement | dropIndexStatement | dropFunctionStat  ent | dropMacroStatement | analyzeStatement | lockStatement | unlockStatement | lockDatabase | unlockDatabase | createRoleStatement | d▽opRoleStatement | grantPrivileges | revokePrivileges | showGrants | showRoleGrants | showRolePrincipals | showRoles | grantRole | revokeRole | setRole | showCurrentRole );])
        at org.antlr.runtime.DFA.noViableAlt(DFA.java:158)
        at org.antlr.runtime.DFA.predict(DFA.java:116)
        at org.apache.hadoop.hive.ql.parse.HiveParser.ddlStatement(HiveParser.java:2084)
        at org.apache.hadoop.hive.ql.parse.HiveParser.execStatement(HiveParser.java:1392)
        at org.apache.hadoop.hive.ql.parse.HiveParser.statement(HiveParser.java:1030)
        at org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:199)
        at org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:166)
        at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:417)
        at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:335)
        at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1026)
        at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1091)
        at org.apache.hadoop.hive.ql.Driver.run(Driver.java:962)
        at org.apache.hadoop.hive.ql.Driver.run(Driver.java:952)
        at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:269)
        at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:221)
        at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:431)
        at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:800)
        at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:694)
        at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:633)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
FAILED: ParseException line 1:5 cannot recognize input near 'show' 'database' '<EOF>' in ddl statement

错误原因：忘加s了

解决方法：将 show database 改成 show databases

坑二：FAILED: HiveException java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStore

错误信息：

FAILED: HiveException java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStore

错误原因：没有初始化元数据数据库

解决办法：schematool -dbType mysql -initSchema 没配置环境变量，去bin下找

该内容执行过程中也会出问题

坑三：这种坑没有错误，要看说明

错误信息：在hive-default.xml.template中

  <!-- WARNING!!! This file is auto generated for documentation purposes ONLY! -->
  <!-- WARNING!!! Any changes you make to this file will be ignored by Hive.   -->
  <!-- WARNING!!! You must make your changes in hive-site.xml instead.         -->

  <!-- 警告!!! 此文件是自动生成的，仅用于文档目的! -->
  <!-- 警告!!! 您对该文件所做的任何更改都将被Hive忽略。   -->
  <!-- 警告!!! 您必须在hive-site.xml中进行更改。        -->

错误原因：没有hive-site.xml

解决办法：cp hive-default.xml.template hive-site.xml

坑四：Caused by: com.ctc.wstx.exc.WstxParsingException: Illegal character entity: expansion character (code 0x8

错误信息：

Exception in thread "main" java.lang.RuntimeException: com.ctc.wstx.exc.WstxParsingException: **Illegal character entity: expansion character (code 0x8
 at [row,col,system-id]: [3210,96,"file:/opt/apache-hive-3.1.1-bin/conf/hive-site.xml"]**
	at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:3003)
	at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2931)
	at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2806)
	at org.apache.hadoop.conf.Configuration.get(Configuration.java:1460)
	at org.apache.hadoop.hive.conf.HiveConf.getVar(HiveConf.java:4990)
	at org.apache.hadoop.hive.conf.HiveConf.getVar(HiveConf.java:5063)
	at org.apache.hadoop.hive.conf.HiveConf.initialize(HiveConf.java:5150)
	at org.apache.hadoop.hive.conf.HiveConf.<init>(HiveConf.java:5098)
	at org.apache.hive.beeline.HiveSchemaTool.<init>(HiveSchemaTool.java:96)
	at org.apache.hive.beeline.HiveSchemaTool.main(HiveSchemaTool.java:1473)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.hadoop.util.RunJar.run(RunJar.java:318)
	at org.apache.hadoop.util.RunJar.main(RunJar.java:232)
Caused by: com.ctc.wstx.exc.WstxParsingException: Illegal character entity: expansion character (code 0x8
 at [row,col,system-id]: [3210,96,"file:/opt/apache-hive-3.1.1-bin/conf/hive-site.xml"]
	at com.ctc.wstx.sr.StreamScanner.constructWfcException(StreamScanner.java:621)
	at com.ctc.wstx.sr.StreamScanner.throwParseError(StreamScanner.java:491)
	at com.ctc.wstx.sr.StreamScanner.reportIllegalChar(StreamScanner.java:2456)
	at com.ctc.wstx.sr.StreamScanner.validateChar(StreamScanner.java:2403)
	at com.ctc.wstx.sr.StreamScanner.resolveCharEnt(StreamScanner.java:2369)
	at com.ctc.wstx.sr.StreamScanner.fullyResolveEntity(StreamScanner.java:1515)
	at com.ctc.wstx.sr.BasicStreamReader.nextFromTree(BasicStreamReader.java:2828)
	at com.ctc.wstx.sr.BasicStreamReader.next(BasicStreamReader.java:1123)
	at org.apache.hadoop.conf.Configuration$Parser.parseNext(Configuration.java:3257)
	at org.apache.hadoop.conf.Configuration$Parser.parse(Configuration.java:3063)
	at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2986)
	... 15 more

错误原因：配置文件中多了一个违法字符&#8

解决办法：vi hive-site.xml 也有可能还没到这一步 vi hive-default.xml

跳到3215行或者3232行，删除掉&#8

坑五：Error: Syntax error: Encountered "<EOF>" at line 1, column 64.

错误信息：

SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/hive/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/hadoop/hadoop-2.7.7/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Metastore connection URL:	 jdbc:derby:;databaseName=metastore_db;create=true
Metastore Connection Driver :	 org.apache.derby.jdbc.EmbeddedDriver
Metastore connection User:	 APP
Starting metastore schema initialization to 2.3.0
Initialization script hive-schema-2.3.0.mysql.sql
Connecting to jdbc:derby:;databaseName=metastore_db;create=true
Connected to: Apache Derby (version 10.10.2.0 - (1582446))
Driver: Apache Derby Embedded JDBC Driver (version 10.10.2.0 - (1582446))
Transaction isolation: TRANSACTION_READ_COMMITTED
0: jdbc:derby:> !autocommit on
Autocommit status: true
0: jdbc:derby:> /*!40101 SET @OLD_CHARACTER_SET_CLIENT=@@CHARACTER_SET_CLIENT */
Error: Syntax error: Encountered "<EOF>" at line 1, column 64. (state=42X01,code=30000)
Closing: 0: jdbc:derby:;databaseName=metastore_db;create=true
org.apache.hadoop.hive.metastore.HiveMetaException: Schema initialization FAILED! Metastore state would be inconsistent !!
Underlying cause: java.io.IOException : Schema script failed, errorcode 2

错误原因：没有指定元数据库位置和连接信息或错误连接信息

网上一些解决办法都是这样的，在hive-site.xml中配置这个信息，但还会出现同样错误

<property>
	<name>javax.jdo.option.ConnectionURL</name>
	<value>jdbc:mysql://hadoop1:3306/hive?useSSL=false</value>
</property>
<property>
	<name>javax.jdo.option.ConnectionDriverName</name>
	<value>com.mysql.jdbc.Driver</value>
</property>
<property>
	<name>javax.jdo.option.ConnectionUserName</name>
	<value>root</value>
</property>
<property>
	<name>javax.jdo.option.ConnectionPassword</name>
	<value>123456</value>
</property>
<property>
	<name>hive.metastore.schema.verification</name>
	<value>false</value>
</property>

解决办法：vi hive-site.xml

按照配置去找 568行，584行，797行，1101行，1126行。

具体找法是进入vi不进入编辑状态，输入行号加G例如568G

坑6：Caused by: java.net.URISyntaxException: Relative path in absolute URI: ${system:java.io.tmpdir%7D/$%7Bsystem:user.name%7D

错误信息：

SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/hadoop/app/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/hadoop/app/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Hive Session ID = 43f139fe-0200-4f89-906b-2fe8109c3b86

Logging initialized using configuration in file:/home/hadoop/app/hive/conf/hive-log4j2.properties Async: true
Exception in thread "main" java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path in absolute URI: ${system:java.io.tmpdir%7D/$%7Bsystem:user.name%7D
        at org.apache.hadoop.fs.Path.initialize(Path.java:259)
        at org.apache.hadoop.fs.Path.<init>(Path.java:217)
        at org.apache.hadoop.hive.ql.session.SessionState.createSessionDirs(SessionState.java:710)
        at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:627)
        at org.apache.hadoop.hive.ql.session.SessionState.beginStart(SessionState.java:591)
        at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:747)
        at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:683)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.hadoop.util.RunJar.run(RunJar.java:318)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:232)
Caused by: java.net.URISyntaxException: Relative path in absolute URI: ${system:java.io.tmpdir%7D/$%7Bsystem:user.name%7D
        at java.net.URI.checkPath(URI.java:1823)
        at java.net.URI.<init>(URI.java:745)
        at org.apache.hadoop.fs.Path.initialize(Path.java:256)
        ... 12 more

错误原因：没有设置数据临时数据位置或临时数据位置解析失败

解决办法：

第一步：在$HOME_HIVE也就是hive家目录创建三个文件夹tmp，logs，download分别存储临时文件，查询日志，下载的资源

cd $HIVE_HOME
mkdir tmp
mkdir logs
mkdir download

第二步：参考下面内容配置参数

<property>
  <name>hive.exec.local.scratchdir</name>
    <value>$HIVE_HOME/tmp</value>
    <description>Local scratch space for Hive jobs</description>
</property>
<property>
   <name>hive.querylog.location</name>
    <value>$HIVE_HOME/logs</value>
    <description>Location of Hive run time structured log file</description>
 </property>
 <property>
   <name>hive.downloaded.resources.dir</name>
    <value>$HIVE_HOME/download</value>
    <description>Temporary local directory for added resources in the remote file system.</description>
 </property>

具体操作：vi hive-site.xml

跳至142行，147行，1845行，修改内容

如果在show databases还出现错误，看下一个

坑7：IllegalArgumentException java.net.URISyntaxException: Relative path in absolute URI:

错误信息：

hive> show databases;
FAILED: IllegalArgumentException java.net.URISyntaxException: Relative path in absolute URI: file:./$HIVE_HOME/tmp/1ef383fc-eff9-475c-bd54-bd87a5d16414/hive_2019-09-03_22-46-44_530_11458349

错误原因：解析不了$HIVE_HOME

解决办法：错误6中的$HIVE_HOME都改成真正对应的路径如/home/hadoop/app/hive即可

坑8：FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask

错误信息：

hive> select * from xixi;
OK
Time taken: 4.745 seconds
hive> insert into xixi values(1);
Query ID = hadoop_20190910153718_1860e7ec-88d5-441b-9854-81e6872ce034
Total jobs = 3
Launching Job 1 out of 3
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
  set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
  set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
  set mapreduce.job.reduces=<number>
Starting Job = job_1567403212809_0004, Tracking URL = http://hadoop1:8088/proxy/application_1567403212809_0004/
Kill Command = /home/hadoop/app/hadoop/bin/mapred job  -kill job_1567403212809_0004
Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 1
2019-09-10 15:38:42,612 Stage-1 map = 0%,  reduce = 0%
2019-09-10 15:38:44,268 Stage-1 map = 100%,  reduce = 100%
Ended Job = job_1567403212809_0004 with errors
Error during job, obtaining debugging information...
Examining task ID: task_1567403212809_0004_m_000000 (and more) from job job_1567403212809_0004

Task with the most failures(4): 
-----
Task ID:
  task_1567403212809_0004_m_000000

URL:
  http://hadoop1:8088/taskdetails.jsp?jobid=job_1567403212809_0004&tipid=task_1567403212809_0004_m_000000
-----
Diagnostic Messages for this Task:
[2019-09-10 15:38:42.526]Container [pid=19303,containerID=container_1567403212809_0004_01_000005] is running 443861504B beyond the 'VIRTUAL' memory limit. Current usage: 64.2 MB of 1 GB physical memory used; 2.5 GB of 2.1 GB virtual memory used. Killing container.
Dump of the process-tree for container_1567403212809_0004_01_000005 :
        |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
        |- 19315 19303 19303 19303 (java) 522 330 2582822912 16141 /data/app/java/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Xmx820m -Djava.io.tmpdir=/home/hadoop/tmp/nm-local-dir/usercache/hadoop/appcache/application_1567403212809_0004/container_1567403212809_0004_01_000005/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/home/hadoop/app/hadoop/logs/userlogs/application_1567403212809_0004/container_1567403212809_0004_01_000005 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA -Dhadoop.root.logfile=syslog org.apache.hadoop.mapred.YarnChild 192.168.85.130 36324 attempt_1567403212809_0004_m_000000_3 5 
        |- 19303 19302 19303 19303 (bash) 0 0 115896320 306 /bin/bash -c /data/app/java/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN   -Xmx820m -Djava.io.tmpdir=/home/hadoop/tmp/nm-local-dir/usercache/hadoop/appcache/application_1567403212809_0004/container_1567403212809_0004_01_000005/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/home/hadoop/app/hadoop/logs/userlogs/application_1567403212809_0004/container_1567403212809_0004_01_000005 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA -Dhadoop.root.logfile=syslog org.apache.hadoop.mapred.YarnChild 192.168.85.130 36324 attempt_1567403212809_0004_m_000000_3 5 1>/home/hadoop/app/hadoop/logs/userlogs/application_1567403212809_0004/container_1567403212809_0004_01_000005/stdout 2>/home/hadoop/app/hadoop/logs/userlogs/application_1567403212809_0004/container_1567403212809_0004_01_000005/stderr  

[2019-09-10 15:38:42.963]Container killed on request. Exit code is 143
[2019-09-10 15:38:43.258]Container exited with a non-zero exit code 143. 


FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
MapReduce Jobs Launched: 
Stage-Stage-1: Map: 1  Reduce: 1   HDFS Read: 0 HDFS Write: 0 FAIL

错误原因：每个map分配的内存过小

解决办法：