CDH6.2.1 hive2.1导入orc分区表异常处理过程

• 1、查询数据

1、Spark-sql查询正常

2、Hsql查询异常

Failed with exception java.io.IOException:java.lang.RuntimeException: ORC split generation failed with exception: java.lang.ArrayIndexOutOfBoundsException: 6

1. spark-shell查询异常

3.1、查询报错

Truncated the string representation of a plan since it was too large. This behavior can be adjusted by setting

‘spark.debug.maxToStringFields’ in SparkEnv.conf.

3.2、无法获取到数据

二、解决异常

1、Hsql查询异常解决

参考链接：

hive2.1.1读取spark写入的orc:ORC split generation failed with exception:ArrayIndexOutOfBoundsException: 6_lixiaoksi的博客-CSDN博客https://blog.csdn.net/lixiaoksi/article/details/106855509

hive-exec-2.1.1-cdh6.1.1.jar_-Hive文档类资源-CSDN下载修改后的hive-exec-2.1.1-cdh6.1.1.jar包更多下载资源、学习资料请访问CSDN下载频道.https://download.csdn.net/download/lixiaoksi/16333490

替换CDH集群的jar包(每个节点都需要替换，注意备份原文件)，重启hive客户端：

/opt/cloudera/parcels/CDH/lib/hive/lib/hive-exec-2.1.1-cdh6.2.1.jar

/opt/cloudera/parcels/CDH/jars/hive-exec-2.1.1-cdh6.2.1.jar

/opt/cloudera/parcels/CDH/lib/spark/hive/hive-exec-2.1.1-cdh6.2.1.jar

1. spark-shell异常解决

参考链接：

Spark: "Truncated the string representation of a plan since it was too large." Warning when using manually created aggregation expression - Stack Overflowhttps://stackoverflow.com/questions/43759896/spark-truncated-the-string-representation-of-a-plan-since-it-was-too-large-w

[SPARK-15705] Spark won't read ORC schema from metastore for partitioned tables - ASF JIRAhttps://issues.apache.org/jira/browse/SPARK-15705

spark-defaults.conf增加配置：

spark.debug.maxToStringFields=200

spark.sql.hive.convertMetastoreOrc=false