Spark常用

一、合并 group by的值
1.concat_ws 连接几个值

concat_ws(',',"test","test1") as product_code,

结果"test,test1"

2.collect_set 去重

concat_ws(',',collect_set(product_code)) as product_code,

3.collect_list 不去重

concat_ws(',',collect_list(product_code)) as product_code

failed的任务可以通过 yarn logs -applicationId $applicationId; $applicationId 是对应的任务id