UDF overloading in spark

没啥好方法，spark注册udf不能重载，只能通过hive udf曲线救国，然后只能用hiveContext了，不能使用sqlContext。。。

UDF are User Defined Function which are register with hive context to use custom functions in spark SQL queries. For example if you want to prepend some string in any other string or column then you can create a following UDF

def addSymbol(input:String, symbol:String)={
symbol+input
}

Now to register above function in hiveContext we need to register UDF as follows

hiveContext.udf.register("addSymbol",(input:String,symbol:String)=>addSymbol(input,symbol))

Now you can use above udf in your sql query in Spark SQL as like below:

hiveContext.sql("select addSymbol('50000','$')").show

Now if you want to overload the above udf for another signature like if user call addSymbol function with single argument and we prepend default String, So now come in your mind is to create another function for addSymbol with single argument add register it with hiveContext like above. Okay once try and then come back you will get your answer.

Its works?

Answer is not, you were see there is no error  when you register another udf with same name but now you cant use first register function signature. Spark hiveContext only register one udf with one name so its register last one so now when you use first signature its give you exception.

So for above problem we have one solution we just create a hive UDF with creating a class which extend hive UDF class as below.

import org.apache.hadoop.hive.ql.exec.UDF

class AddSymbol extends UDF {

  def evaluate(input:String, symbol:String): String = {
    symbol+input
  }
  def evaluate(input:String): String = {
    "$"+input
  }
}

In above class you can overload evaluate function to overload different signatures.

Next you need to create hive temporary function with the above class as follows:

hiveContext.sql("CREATE TEMPORARY FUNCTION addSymbol AS 'AddSymbol'")

Now the above function addSymbol is available as a temporary function of hive, you see as below we can call both the signature of UDF.

hiveContext.sql("select addSymbol('5000','&')").show
hiveContext.sql("select addSymbol('5000')").show