spark sql 类型转换array,将两个Array [string]类型的spark sql列合并到新的Array [string]列中...

I have two columns in a Spark SQL DataFrame with each entry in either column as an array of strings.

val ngramDataFrame = Seq(

(Seq("curious", "bought", "20"), Seq("iwa", "was", "asj"))

).toDF("filtered_words", "ngrams_array")

I want to merge the arrays in each row to make a single array in a new column. My code is as follows:

def concat_array(firstarray: Array[String],

secondarray: Array[String]) : Array[String] =

{ (firstarray ++ secondarray).toArray }

val concatUDF = udf(concat_array _)

val concatFrame = ngramDataFrame.withColumn("full_array", concatUDF($"filtered_words", $"ngrams_array"))

I can successfully use the concat_array function on two arrays. However when I run the above code, I get the followin