mysql collect_set_使用collect_list和collect_set在SQL星火

According to the docs, the collect_set and collect_list functions should be available in Spark SQL. However, I cannot get it to work. I'm running Spark 1.6.0 using a Docker image.

I'm trying to do this in Scala:

import org.apache.spark.sql.functions._

df.groupBy("column1")

.agg(collect_set("column2"))

.show()

And receive the following error at runtime:

Exception in thread "main" org.apache.spark.sql.AnalysisException: undefined function collect_set;

Also tried it using pyspark, but it also fails. The docs state these functions are aliases of Hive UDAFs, but I can't figure out to enable these functions.

How to fix this? Thanx!

解决方案

Spark 2.0+:

You have to enable Hive support for a given SparkSession:

In Scala:

val spark = SparkSession.builder

.master("local")

.appName("testing")

.enableHiveSupport() //

.getOrCreate()

In Python:

spark = (SparkSession.builder

.enableHiveSupport()

.getOrCreate())

Spark < 2.0:

To be able to use Hive UDFs you have use Spark built with Hive support (this is already covered when you use pre-built binaries what seems to be the case here) and initialize SparkContext using HiveContext.

In Scala:

import org.apache.spark.sql.hive.HiveContext

import org.apache.spark.sql.SQLContext

val sqlContext: SQLContext = new HiveContext(sc)

In Python:

from pyspark.sql import HiveContext

sqlContext = HiveContext(sc)