scala implicit - implicit parameters spark 应用

 

语法

class PreferredPrompt(val preference: String)

object Greeter {

   def greet(name:String)(implicit prompt: PreferredPrompt) {

     println("Welcome, " + name + ". Thee systeme is ready.")  
     println(prompt.preference)
   }
}

implicit val prompt = new PreferredPrompt("Yes, master> ")

Greeter.greet("Joe")

 

应用

RequestContext 传递

java编程中有使用ThreadLocal来保存当前请求，避免每个方法都带上context；

scala通过implict，减少boilerplate code 

abstract class RequestHandler {
      def process(context:RequstContext)
}

//main handler class
def dispatch(request:Request) {
   implicit val context = makeContext(request)

   handler1.process
   handler2.process
   handler3.process

}

 

SparkSession

spark session 对象贯穿整个job/application，通过implicit 减少每个方法调用时显示传入

 

  def sql2df(sql:String, view:Option[String] = None, isCache:Boolean = true)(implicit spark:SparkSession):DataFrame = {
    println(sql)
    val df = spark.sql(sql)

    if(isCache)
      df.cache()

    if(view.isDefined)
      df.createOrReplaceTempView(view.get)

    df
  }