前几天将hive的版本由 0.8.1升级到 0.11.0 ,新版本新增了很多内置函数,执行效率比之前也有了一定的提升,
但是有新的问题产生,问题如下:
create table tab1
(
id int,
name string
);
create table tab2
(
id int
);
select a.name,count(*)
from tab1 a left outer join tab2 b on a.id = b.id
group by a.name;
执行上面的语句的时候
会报错,错误信息:
Diagnostic Messages for this Task:
java.lang.RuntimeException: Hive Runtime Error while closing operators: java.lang.IllegalArgumentException: SequenceFile doesn't work with GzipCodec without native-hadoop code!
at org.apache.hadoop.hive.ql.exec.ExecReducer.close(ExecReducer.java:313)
at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:479)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:417)
at org.apache.hadoop.mapred.Child$4.run(Child.java:266)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1278)
at org.apache.hadoop.mapred.Child.main(Child.java:260)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.IllegalArgumentException: SequenceFile doesn't work with GzipCodec without native-hadoop code!
at org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getHiveRecordWriter(HiveFileFormatUtils.java:240)
at org.apache.hadoop.hive.ql.exec.FileSink
FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask
通过对比发现,有个配置参数的默认值是不同的。
hive.exec.compress.intermediate=false 0.8.1
hive.exec.compress.intermediate=ture 0.11.0
将这个参数修改为false之后 这个问题就修复了。