spark读取hive表字段，区分大小写问题

最新推荐文章于 2024-07-20 22:06:32 发布

MrZhangBaby

最新推荐文章于 2024-07-20 22:06:32 发布

阅读量1.2k

点赞数

文章标签： spark hive 大数据

本文链接：https://blog.csdn.net/mrzhangbaby/article/details/133793660

版权

背景

spark任务读取hive表，查询字段为小写，但Hive表字段为大写，无法读取数据

问题错误:

如何解决呢？

In version 2.3 and earlier, when reading from a Parquet data source table, Spark always returns null for any column whose column names in Hive metastore schema and Parquet schema are in different letter cases, no matter whether spark.sql.caseSensitive is set to true or false. Since 2.4, when spark.sql.caseSensitive is set to false, Spark does case insensitive column name resolution between Hive metastore schema and Parquet schema, so even column names are in different letter cases, Spark returns corresponding column values. An exception is thrown if there is ambiguity, i.e. more than one Parquet column is matched. This change also applies to Parque