如果sparkStreaming运行在windows本地(笔者OS 为 win10), 大概率会报如下错误, 相应的可能的解决方案如下:
Q1. 缺包报错
sing Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
Exception in thread "main" java.lang.NoClassDefFoundError: scala/xml/MetaData
at org.apache.spark.ui.jobs.JobsTab.<init>(JobsTab.scala:45)
at org.apache.spark.ui.SparkUI.initialize(SparkUI.scala:61)
at org.apache.spark.ui.SparkUI.<init>(SparkUI.scala:80)
at org.apache.spark.ui.SparkUI$.create(SparkUI.scala:175)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:443)
at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2486)
at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:930)
at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:921)
at scala.Option.getOrElse(Option.scala:121)
at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:921)
at com.ssic.report.BigScreenModel$.main(BigScreenModel.scala:50)
at com.ssic.report.BigScreenModel.main(BigScreenModel.scala)
Caused by: java.lang.ClassNotFoundException: scala.xml.MetaData
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:355)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
... 12 more
A1. 增加缺失的 scala-xml jar包
pom.xml中添加scala-xml jar包支持.
<dependency>
<groupId>org.scala-lang</groupId>
<artifactId>scala-xml</artifactId>
<version>2.11.0-M4</version>
</dependency>
Q2. 本地路径需为file:///格式
A2. 由于程序运行在windows本地, 分布式文件系统也跑在本地, 需要将 HDFS路径中的hdfs:///需要修改为file:///, 如下所示
//SparkStreaming程序临时停止的HDFS标志文件
val osType = SparkDebugTools.getOSType()
var hdfsPrefix = "hdfs://"
if ("linux".equals(osType)) hdfsPrefix else hdfsPrefix = "file://"
val HDFS_SHUTDOWN_MARKER = hdfsPrefix + "/tmp/streaming_stop_watermark/stop_sign"
Q3. Win或Mac开发环境下开发调试配置
这种环境下,需要程序在本地启动,可以通过如下方式实现:
①、开发工具类
import org.apache.spark.SparkConf
/**
* Spark调试工具类
*
* 这里默认大家的开发环境为Windows 或 Mac 操作系统,而生产发布使用Linux。
* 如果不是,请自行适配。
*
* 如果是 Windows 或 Mac 操作系统,则允许用户在本地进行调试.
*/
object SparkDebugTools {
private val os = System.getProperty( "os.name" ).toLowerCase
def getOSType(): String = {
System.getProperty( "os.name" ).toLowerCase
}
def tryEnableLocalRun(sparkConf: SparkConf): Unit = {
//if (os.indexOf( "windows" ) >= 0 || os.indexOf( "mac" ) >= 0) {
if (os.indexOf( "linux" ) < 0) {
sparkConf.setMaster( "local[1]" )
}
}
}
②、Spark SSC对象构建时引入setMaster功能
val sparkConf = new SparkConf().setAppName("MyDemo")
sparkConf.set("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
sparkConf.set("spark.streaming.kafka.maxRatePerPartition", "200")
sparkConf.set("spark.driver.allowMultipleContexts", "true")
sparkConf.set("spark.debug.maxToStringFields", "200")
sparkConf.set("spark.streaming.stopGracefullyOnShutdown", "true") //支撑ssc优雅的关闭
SparkDebugTools.tryEnableLocalRun(sparkConf)
val session: SparkSession = SparkSession.builder().config(sparkConf).getOrCreate()
val sc = new SparkContext(sparkConf)
val ssc = new StreamingContext(session.sparkContext, Seconds(5))