记一次hive查询报错解决 No enum constant org.apache.parquet.hadoop.metadata.CompressionCodecName.LZOP

背景

在正常开发中建表进行数据插入测试,突然报错
代码如下:

INSERT OVERWRITE TABLE dim_common.dim_common_product_..._sku
SELECT 
   id,
  .....
FROM ods_common.ods_common_product_..._sku

建表语句:

CREATE EXTERNAL TABLE dim_common.dim_common_product_..._sku(
  `id` bigint COMMENT '主键',
......)
COMMENT 
STORED AS parquet
LOCATION '/big-data/dim/common/dim_common_product_..._sku'
TBLPROPERTIES ('parquet.compression'='lzop');

报错如下

Error: java.lang.RuntimeException: Hive Runtime Error while closing operators
	at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:211)
	at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61)
	at org.apache.hadoop.hive.ql.exec.mr.ExecMapRunner.run(ExecMapRunner.java:37)
	at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:465)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:349)
	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.IllegalArgumentException: No enum constant org.apache.parquet.hadoop.metadata.CompressionCodecName.LZOP
	at org.apache.hadoop.hive.ql.exec.FileSinkOperator.createBucketFiles(FileSinkOperator.java:742)
	at org.apache.hadoop.hive.ql.exec.FileSinkOperator.closeOp(FileSinkOperator.java:1260)
	at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:733)
	at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:757)
	at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:757)
	at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:757)
	at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:193)
	... 9 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.IllegalArgumentException: No enum constant org.apache.parquet.hadoop.metadata.CompressionCodecName.LZOP
	at org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getHiveRecordWriter(HiveFileFormatUtils.java:285)
	at org.apache.hadoop.hive.ql.exec.FileSinkOperator.createBucketForFileIdx(FileSinkOperator.java:780)
	at org.apache.hadoop.hive.ql.exec.FileSinkOperator.createBucketFiles(FileSinkOperator.java:731)
	... 15 more
Caused by: java.lang.IllegalArgumentException: No enum constant org.apache.parquet.hadoop.metadata.CompressionCodecName.LZOP
	at java.lang.Enum.valueOf(Enum.java:238)
	at org.apache.parquet.hadoop.metadata.CompressionCodecName.valueOf(CompressionCodecName.java:26)
	at org.apache.parquet.hadoop.metadata.CompressionCodecName.fromConf(CompressionCodecName.java:39)
	at org.apache.hadoop.hive.ql.io.parquet.write.ParquetRecordWriterWrapper.initializeSerProperties(ParquetRecordWriterWrapper.java:119)
	at org.apache.hadoop.hive.ql.io.parquet.write.ParquetRecordWriterWrapper.<init>(ParquetRecordWriterWrapper.java:65)
	at org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat.getParquerRecordWriterWrapper(MapredParquetOutputFormat.java:137)
	at org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat.getHiveRecordWriter(MapredParquetOutputFormat.java:126)
	at org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getRecordWriter(HiveFileFormatUtils.java:297)
	at org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getHiveRecordWriter(HiveFileFormatUtils.java:282)
	... 17 more


FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
MapReduce Jobs Launched:
Stage-Stage-1: Map: 2  Reduce: 1   HDFS Read: 0 HDFS Write: 0 FAIL
Total MapReduce CPU Time Spent: 0 msec

报错分析:

首先去掉了insert语句测试了查询语句发现正常执行,排除了查询语句错误。
添加insert之后报上错误,因此推测是集群压缩配置或者建表压缩配置有问题,因为并不是第一次建表,其他表均无异常。所以详细检查了建表语句最终发现压缩配置错误。
错误配置:

TBLPROPERTIES ('parquet.compression'='lzop');

修改为正确配置后解决:

TBLPROPERTIES ('parquet.compression'='lzo');

这里仅此小记,总结粗心教训,为大家解决问题提供思路。

  • 1
    点赞
  • 1
    收藏
    觉得还不错? 一键收藏
  • 打赏
    打赏
  • 1
    评论

“相关推荐”对你有帮助么?

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论 1
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包

打赏作者

扫地增

你的鼓励将是我创作的最大动力

¥1 ¥2 ¥4 ¥6 ¥10 ¥20
扫码支付:¥1
获取中
扫码支付

您的余额不足,请更换扫码支付或充值

打赏作者

实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值