[解决办法] Cannot have map type columns in DataFrame which calls set operations(intersect, except, etc.)

[解决办法] Sql执行错误:org.apache.spark.sql.AnalysisException: Cannot have map type columns in DataFrame which calls set operations(intersect, except, etc.), but the type of column extend_value is map<string,string>

原始语句

select distinct id
	, cols
	, extend_value 
from test.test_table 

报错信息

Sql执行错误:org.apache.spark.sql.AnalysisException: Cannot have map type columns in DataFrame which calls set operations(intersect, except, etc.), but the type of column extend_value is map<string,string>;; 

尝试 group by 解决

select id
	, cols
	, extend_value 
from test.test_table
group by id, cols, extend_value

依旧报错

Sql执行错误:org.apache.spark.sql.AnalysisException: expression test_table.`extend_value` cannot be used as a grouping expression because its data type map<string,string> is not an orderable data type.;; 

解决方案

方案一 分片排序后取第一条

select id
	, cols
	, extend_value 
from (
	select id
		, cols
		, extend_value 
		, row_number() over (partition by id, order by cols) as row_num
	from test.test_table )
where row_num = 1

方案二（待加入）