java dataframe 遍历,如何在GroupBy操作后从spark DataFrame列中收集字符串列表？

The solution described here (by zero323) is very close to what I want with two twists:

How do I do it in Java?

What if the column had a List of Strings instead of a single String and I want to collect all such lists into a single list after GroupBy(some other column)?

I am using Spark 1.6 and have tried to use

org.apache.spark.sql.functions.collect_list(Column col) as described in the solution to that question, but got the following error

Exception in thread "main" org.apache.spark.sql.AnalysisException: undefined function collect_list;

at org.apache.spark.sql.catalyst.analysis.SimpleFunctionRegistry$$anonfun$2.apply(FunctionRegistry.scala:65)

at org.apache.spark.sql.catalyst.analysis.SimpleFunctionRegistry$$anonfun$2.apply(FunctionRegistry.scala:65)

at scala.Option.getOrElse(Option.scala:121)

解决方案

Error you see suggests you use plain SQLContext not HiveContext. collect_list is a Hive UDF and as such requires HiveContext. It also doesn't support complex columns so the only option is to explode first:

import org.apache.spark.api.java.*;

import org.apache.spark.SparkConf;

import org.apache.spark.sql.SQLContext;

import org.apache.spark.sql.hive.HiveContext;

import java.util.*;

import org.apache.spark.sql.DataFrame;

import static org.apache.spark.sql.functions.*;

public class App {

public static void main(String[] args) {

JavaSparkContext sc = new JavaSparkContext(new SparkConf());

SQLContext sqlContext = new HiveContext(sc);

List data = Arrays.asList(

"{\"id\": 1, \"vs\": [\"a\", \"b\"]}",

"{\"id\": 1, \"vs\": [\"c\", \"d\"]}",

"{\"id\": 2, \"vs\": [\"e\", \"f\"]}",

"{\"id\": 2, \"vs\": [\"g\", \"h\"]}"

);

DataFrame df = sqlContext.read().json(sc.parallelize(data));

df.withColumn("vs", explode(col("vs")))

.groupBy(col("id"))

.agg(collect_list(col("vs")))

.show();

}

}

It is rather unlikely it will perform well though.