Spark StreamingContext创建方式

方式1

//get a conf  object
  val conf = new SparkConf().setAppName("SparkStreamWordcount").setMaster("local[2]")
  val duration: Duration = Seconds.apply(10)
  //获取StreamingContext
  val ssc = new StreamingContext(conf, duration)
  val dstream: ReceiverInputDStream[String] = ssc.socketTextStream("mypc01", 10086)

方式二

val spark: SparkSession = SparkSession.builder().appName("test").master("local[2]").getOrCreate()
  val duration: Duration = Seconds.apply(10)
  //通过sparkContext构建StreamingContext
  val ssc = new StreamingContext(spark.sparkContext, duration)
  val dstream: ReceiverInputDStream[String] = ssc.socketTextStream("mypc", 10086)

方式三

 val spark: SparkSession = SparkSession.builder().appName("test").master("local[2]").getOrCreate()
  val duration: Duration = Seconds.apply(10)
  //也可以通过sparkConf构建StreamingContext
  val ssc = new StreamingContext(spark.sparkContext.getConf, duration)
  val dstream: ReceiverInputDStream[String] = ssc.socketTextStream("mypc01", 10086)

报错

Only one SparkContext may be running in this JVM (see SPARK-2243). To
ignore this error, set spark.driver.allowMultipleContexts = true. The
currently running SparkContext was created at:
org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:910)

总结

• 通过sparkContext 或者SparkConf都可以创建StreamingContext
• 最好的方式是SparkSession 包含StreamingContext 不过目前官方还没有实现