spark sql 异常

最新推荐文章于 2024-05-01 02:13:07 发布

浩海紫冰

最新推荐文章于 2024-05-01 02:13:07 发布

阅读量1.2k

点赞数

分类专栏：大数据文章标签： spark sql

本文链接：https://blog.csdn.net/stay_forever/article/details/108218702

版权

大数据专栏收录该内容

14 篇文章 0 订阅

订阅专栏

1.spark SQL 测试过程中报如下错误 org.apache.spark.sql.AnalysisException

--

20/08/25 11:42:08 INFO Client: resolveAppExceptionMsg, msg start
20/08/25 11:42:08 INFO Client: resolveAppExceptionMsg, msg:User class threw exception: org.apache.spark.sql.AnalysisException: expression 'dpstg_dpmid_platform_shopping_mtshop_coop_online_d_v0_20200824.`event_type`' is neither present in the group by, nor is it an aggregate function. Add to group by or wrap in first() (or first_value) if you don't care which value you get.;;
Project [moudle2moudle AS mark#183, moudle2moudle AS revisit_type#184, platform_id#219, all AS bu_class_id#185, all AS bu_class_name#186, page_identifier#222, page_name#223, all AS event_id#187, element_name#161 AS module_name#188, module_product_name#225, view_pv#175L, view_uv#176L, tap_uv#177L, tap_pv#178L, revisit_ratio_7d#179, revisit_ratio_30d#180, revisit_click_ratio_7d#181, revisit_click_ratio_30d#182, hp_cal_dt#218]

错误代码如下

SELECT dpid,hp_cal_dt,platform_id,event_type, sum(pv) as pv
FROM shop_table 
group by  dpid,hp_cal_dt,platform_id

修改后：


SELECT dpid,hp_cal_dt,platform_id,event_type, sum(pv) as pv
FROM shop_table 
group by dpid,hp_cal_dt,platform_id,event_type

原因是：上述业务逻辑比较复杂，因此把逻辑进行精简说明主要问题，group by 的维度比较多，grouping sets 部分都省略了，主要挑选dpid,hp_cal_dt,platform_id,event_type维度说明主要问题，org.apache.spark.sql.AnalysisException这种错误的主要原因是语法错误，具体是聚合维度和group by 的维度个数不一致。