spark sql 异常

 

1.spark SQL 测试过程中报如下错误 org.apache.spark.sql.AnalysisException

--

20/08/25 11:42:08 INFO Client: resolveAppExceptionMsg, msg start
20/08/25 11:42:08 INFO Client: resolveAppExceptionMsg, msg:User class threw exception: org.apache.spark.sql.AnalysisException: expression 'dpstg_dpmid_platform_shopping_mtshop_coop_online_d_v0_20200824.`event_type`' is neither present in the group by, nor is it an aggregate function. Add to group by or wrap in first() (or first_value) if you don't care which value you get.;;
Project [moudle2moudle AS mark#183, moudle2moudle AS revisit_type#184, platform_id#219, all AS bu_class_id#185, all AS bu_class_name#186, page_identifier#222, page_name#223, all AS event_id#187, element_name#161 AS module_name#188, module_product_name#225, view_pv#175L, view_uv#176L, tap_uv#177L, tap_pv#178L, revisit_ratio_7d#179, revisit_ratio_30d#180, revisit_click_ratio_7d#181, revisit_click_ratio_30d#182, hp_cal_dt#218]

 错误代码如下

SELECT dpid,hp_cal_dt,platform_id,event_type, sum(pv) as pv
FROM shop_table 
group by  dpid,hp_cal_dt,platform_id

修改后： 

SELECT dpid,hp_cal_dt,platform_id,event_type, sum(pv) as pv
FROM shop_table 
group by dpid,hp_cal_dt,platform_id,event_type
                

原因是：上述业务逻辑比较复杂，因此把逻辑进行精简说明主要问题，group by  的维度比较多，grouping  sets  部分都省略了，主要挑选dpid,hp_cal_dt,platform_id,event_type维度说明主要问题，org.apache.spark.sql.AnalysisException这种错误的主要原因是语法错误，具体是聚合维度和group by 的维度个数不一致。

 

相关链接：https://www.jianshu.com/p/c131657a5813