Spark udf合并两个Array / List / Seq为一个Array / List / Seq,将所有元素放到一个Array / List / Seq中

在写Spark代码时, 发现有个场景需要合并两个List / Array / Seq为一个:
即: 合并两个Array[String]

输入: Array("1","2","3","4") Array("5","6","7","8")
输出: Array("1","2","3","4","5","6","7","8")

或者:

输入: ["1","2","3","4"] ["5","6","7","8"]
输出: ["1","2","3","4","5","6","7","8"]

但是Spark中没有实现类似功能的算子, 于是自己写了一个UDF去实现, 话不多说, 贴代码供大家参考.

UDF代码:

object UdfUtils {
  def udfUnionString(): UserDefinedFunction = {
    def unionString(arrayOne: Seq[String], arrayTwo: Seq[String]): Array[String] = {
      if ((arrayOne == null || arrayOne.isEmpty) && (arrayTwo == null || arrayTwo.isEmpty)) {
        Array[String]()
      } else if (arrayOne == null || arrayOne.isEmpty) {
        arrayTwo.toArray
      } else if (arrayTwo == null || arrayTwo.isEmpty) {
        arrayOne.toArray
      } else {
        (arrayOne ++ arrayTwo).toArray
      }
    }

    udf((arrayOne: Seq[String], arrayTwo: Seq[String]) =>
      unionString(arrayOne, arrayTwo),
      ArrayType(StringType)
    )
  }
}

调用UDF代码:

testDF.select(
    UdfUtils.udfUnionString()(testDf1("array1"),testDf2("array2"))
)

示例:

testDf1("array1"):
    ["1","2","3","4"] 
    ["a1","b1","c1","d1"]
    ...

testDf2("array2"):
    ["5","6","7","8"] 
    ["a2","b2","c2","d2"]
    ...
    
结果:
    ["1","2","3","4","5","6","7","8"]
    ["a1","b1","c1","d1","a2","b2","c2","d2"]