问题:
hive中查询一段语句:
select
guid,
sessionid,
properties,
first_value(properties['pageid']) over(partition by guid,sessionid order by ts) as in_page,
last_value(properties['pageid']) over(partition by guid,sessionid order by ts rows between unbounded preceding and unbounded following) as out_page,
first_value(province) over(partition by guid,sessionid order by ts) as province,
first_value(city) over(partition by guid,sessionid order by ts) as city,
first_value(region) over(partition by guid,sessionid order by ts) as region,
first_value(devicetype) over(partition by guid,sessionid order by ts) as devicetype
from dwd.event_app_detail where dt='2020-09-01' and eventid='pageView'
查询后报错
表数据:
查看表信息,只有一个字段是map类型的,其他字段都是基本类型,且存储格式发现是parquet
show create table dwd.event_app_detail;
查看日志:
排查后发现是hive中的向量化引擎对parquent的优化导致,部分复杂数据类型不是很兼容,会有这种bug
解决方案:
1,把表重新导成orc格式,hive对orc格式兼容比较好
2,关闭向量化引擎优化
set hive.vectorized.execution.enabled=false;