开发本地环境--支撑sparkStreaming开发调试

如果sparkStreaming运行在windows本地(笔者OS 为 win10), 大概率会报如下错误, 相应的可能的解决方案如下:

Q1. 缺包报错

sing Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
Exception in thread "main" java.lang.NoClassDefFoundError: scala/xml/MetaData
	at org.apache.spark.ui.jobs.JobsTab.<init>(JobsTab.scala:45)
	at org.apache.spark.ui.SparkUI.initialize(SparkUI.scala:61)
	at org.apache.spark.ui.SparkUI.<init>(SparkUI.scala:80)
	at org.apache.spark.ui.SparkUI$.create(SparkUI.scala:175)
	at org.apache.spark.SparkContext.<init>(SparkContext.scala:443)
	at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2486)
	at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:930)
	at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:921)
	at scala.Option.getOrElse(Option.scala:121)
	at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:921)
	at com.ssic.report.BigScreenModel$.main(BigScreenModel.scala:50)
	at com.ssic.report.BigScreenModel.main(BigScreenModel.scala)
Caused by: java.lang.ClassNotFoundException: scala.xml.MetaData
	at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:355)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
	... 12 more

A1. 增加缺失的 scala-xml jar包

pom.xml中添加scala-xml jar包支持.

<dependency>
	<groupId>org.scala-lang</groupId>
    <artifactId>scala-xml</artifactId>
    <version>2.11.0-M4</version>
</dependency>

Q2. 本地路径需为file:///格式

A2. 由于程序运行在windows本地, 分布式文件系统也跑在本地, 需要将 HDFS路径中的hdfs:///需要修改为file:///, 如下所示

//SparkStreaming程序临时停止的HDFS标志文件
  val osType = SparkDebugTools.getOSType()
  var hdfsPrefix = "hdfs://"
  if ("linux".equals(osType)) hdfsPrefix else hdfsPrefix = "file://"
  val HDFS_SHUTDOWN_MARKER = hdfsPrefix + "/tmp/streaming_stop_watermark/stop_sign"

Q3. Win或Mac开发环境下开发调试配置

这种环境下，需要程序在本地启动，可以通过如下方式实现：

①、开发工具类

import org.apache.spark.SparkConf

/**
 * Spark调试工具类
 *
 * 这里默认大家的开发环境为Windows 或 Mac 操作系统，而生产发布使用Linux。
 * 如果不是，请自行适配。
 * 
 * 如果是 Windows 或 Mac 操作系统,则允许用户在本地进行调试.
 */
object SparkDebugTools {

  private val os = System.getProperty( "os.name" ).toLowerCase

  def getOSType(): String = {
    System.getProperty( "os.name" ).toLowerCase
  }

  def tryEnableLocalRun(sparkConf: SparkConf): Unit = {
    //if (os.indexOf( "windows" ) >= 0 || os.indexOf( "mac" ) >= 0) {
    if (os.indexOf( "linux" ) < 0) {
      sparkConf.setMaster( "local[1]" )
    }
  }

}

②、Spark SSC对象构建时引入setMaster功能

val sparkConf = new SparkConf().setAppName("MyDemo")
    sparkConf.set("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
    sparkConf.set("spark.streaming.kafka.maxRatePerPartition", "200")
    sparkConf.set("spark.driver.allowMultipleContexts", "true")
    sparkConf.set("spark.debug.maxToStringFields", "200")
    sparkConf.set("spark.streaming.stopGracefullyOnShutdown", "true") //支撑ssc优雅的关闭

    SparkDebugTools.tryEnableLocalRun(sparkConf)

    val session: SparkSession = SparkSession.builder().config(sparkConf).getOrCreate()
	
	val sc = new SparkContext(sparkConf)

    val ssc = new StreamingContext(session.sparkContext, Seconds(5))