CDH6.2.1 hive2.1导入orc分区表异常处理过程

Spark-sql查询正常

 

Hsql查询异常

Failed with exception java.io.IOException:java.lang.RuntimeException: ORC split generation failed with exception: java.lang.ArrayIndexOutOfBoundsException: 6

spark-shell查询异常

查询报错

Truncated the string representation of a plan since it was too large. This behavior can be adjusted by setting

‘spark.debug.maxToStringFields’ in SparkEnv.conf.

无法获取到数据

 

 Hsql查询异常解决

参考链接：

hive2.1.1读取spark写入的orc:ORC split generation failed with exception:ArrayIndexOutOfBoundsException: 6_beetle_lzk的博客-CSDN博客

替换CDH集群的jar包(每个节点都需要替换，注意备份原文件)，重启hive客户端：

/opt/cloudera/parcels/CDH/lib/hive/lib/hive-exec-2.1.1-cdh6.2.1.jar

/opt/cloudera/parcels/CDH/jars/hive-exec-2.1.1-cdh6.2.1.jar

/opt/cloudera/parcels/CDH/lib/spark/hive/hive-exec-2.1.1-cdh6.2.1.jar

spark-shell异常解决

参考链接：

Spark: "Truncated the string representation of a plan since it was too large." Warning when using manually created aggregation expression - Stack Overflow

[SPARK-15705] Spark won't read ORC schema from metastore for partitioned tables - ASF JIRA

spark-defaults.conf增加配置：

spark.debug.maxToStringFields=200

spark.sql.hive.convertMetastoreOrc=false