HIVE1.1.0报错记录

运行环境:


HADOOP 2.2.0

ZOOKEEPER 3.4.6

HIVE 1.1.0


现象:


配置好hive-site.xml后执行Create Table正常,执行insert报错,日志如下:



Exception in thread "main" java.lang.NoSuchMethodError: org.apache.hadoop.hdfs.DFSClient.getKeyProvider()Lorg/apache/hadoop/crypto/key/KeyProvider;

at org.apache.hadoop.hive.shims.Hadoop23Shims$HdfsEncryptionShim.<init>(Hadoop23Shims.java:1152)

at org.apache.hadoop.hive.shims.Hadoop23Shims.createHdfsEncryptionShim(Hadoop23Shims.java:1279)

at org.apache.hadoop.hive.ql.session.SessionState.getHdfsEncryptionShim(SessionState.java:392)

at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.isPathEncrypted(SemanticAnalyzer.java:1756)

at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getStagingDirectoryPathname(SemanticAnalyzer.java:1875)

at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getMetaData(SemanticAnalyzer.java:1689)

at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getMetaData(SemanticAnalyzer.java:1427)

at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genResolvedParseTree(SemanticAnalyzer.java:10132)

at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10147)

at org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:192)

at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:222)

at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:421)

at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:307)

at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1112)

at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1160)

at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1049)

at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1039)

at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:207)

at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:159)

at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:370)

at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:754)

at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:675)

at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:615)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:606)

at org.apache.hadoop.util.RunJar.main(RunJar.java:212)


问题原因:
HIVE 1.1.0兼容性问题,需要修改源码重编译后修复问题详见https://issues.apache.org/jira/browse/HIVE-9957


修改源码详情:
diff --git a/shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java b/shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java
index 7c1887a20d53abd92bca8cb5f9b78030a38f9657..82833a33876e25bf75ae5dad0533c4c4a4ab6cbc 100644
--- a/shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java
+++ b/shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java
@@ -1281,7 +1281,12 @@ private int compareKeyStrength(String keyname1, String keyname2) throws IOExcept
   public HadoopShims.HdfsEncryptionShim createHdfsEncryptionShim(FileSystem fs, Configuration conf) throws IOException {
     URI uri = fs.getUri();
     if ("hdfs".equals(uri.getScheme())) {
-      return new HdfsEncryptionShim(uri, conf);
+      try {	
+        return new HdfsEncryptionShim(uri, conf);
+      } catch (NoSuchMethodError e) {
+        // ignore error as encryption is not supported.
+        // let this method return the unsupported encryption shim instead
+      }
     }
     return new HadoopShims.NoopHdfsEncryptionShim();
   }
删除 "-" 后面的内容
增加 "+"后面的内容


重编译步骤:

1.svn co http://svn.apache.org/repos/asf/hive/trunk hive  //下载源码
  
2.根据上述修改源码详情内容修改 hive/shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java

3.CD hive
  mvn clean install -Phadoop-2,dist
4.cd packaging/target/apache-hive-{version}-SNAPSHOT-bin/apache-hive-{version}-SNAPSHOT-bin//编译好后存放目录



评论 2
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值