Only one SparkContext may be running in this JVM

最新推荐文章于 2023-06-01 14:11:55 发布

weixin_34348111

最新推荐文章于 2023-06-01 14:11:55 发布

阅读量328

点赞数

文章标签：大数据 scala java

原文链接：https://my.oschina.net/u/2000675/blog/848664

版权

2019独角兽企业重金招聘Python工程师标准>>>

当spark streaming和spark sql一起使用时我们要注意只能有一个SparkContext，否则就会报如下错误：

Exception in thread "main" org.apache.spark.SparkException: Only one SparkContext may be running in this JVM (see SPARK-2243). To ignore this error, set spark.driver.allowMultipleContexts = true. The currently running SparkContext was created at:
org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:58)
com.ecp.data_cleaning.Main.main(Main.java:72)
	at org.apache.spark.SparkContext$$anonfun$assertNoOtherContextIsRunning$2.apply(SparkContext.scala:2223)
	at org.apache.spark.SparkContext$$anonfun$assertNoOtherContextIsRunning$2.apply(SparkContext.scala:2219)
	at scala.Option.foreach(Option.scala:257)
	at org.apache.spark.SparkContext$.assertNoOtherContextIsRunning(SparkContext.scala:2219)
	at org.apache.spark.SparkContext$.markPartiallyConstructed(SparkContext.scala:2292)
	at org.apache.spark.SparkContext.<init>(SparkContext.scala:86)
	at org.apache.spark.streaming.StreamingContext$.createNewSparkContext(StreamingContext.scala:836)
	at org.apache.spark.streaming.StreamingContext.<init>(StreamingContext.scala:84)
	at org.apache.spark.streaming.api.java.JavaStreamingContext.<init>(JavaStreamingContext.scala:138)
	at com.ecp.data_cleaning.Main.main(Main.java:74)

我们可以通过StreamingContext去获取SparkContext

JavaStreamingContext jsc = new JavaStreamingContext(sparkconf,new Duration(co.getDuration()));
JavaSparkContext sc = jsc.sparkContext();
SQLContext sqlcontext = new SQLContext(sc);

转载于:https://my.oschina.net/u/2000675/blog/848664