java 获取dataset_例外情况java.util.NoSuchElementException: 无。获取在Spark Dataset save（）操作中...

我有个例外”java.util.NoSuchElementException: 无。获取“当我试图将 DataSet 作为拼花保存到s3存储时：

例外情况：

java.lang.IllegalStateException: Failed to execute CommandLineRunner

at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:787)

at org.springframework.boot.SpringApplication.callRunners(SpringApplication.java:768)

at org.springframework.boot.SpringApplication.run(SpringApplication.java:322)

at org.springframework.boot.SpringApplication.run(SpringApplication.java:1226)

at org.springframework.boot.SpringApplication.run(SpringApplication.java:1215)

...

Caused by: java.util.NoSuchElementException: None.get

at scala.None$.get(Option.scala:347)

at scala.None$.get(Option.scala:345)

at org.apache.spark.sql.execution.datasources.BasicWriteJobStatsTracker$.metrics(BasicWriteStatsTracker.scala:173)

at org.apache.spark.sql.execution.command.DataWritingCommand$class.metrics(DataWritingCommand.scala:51)

at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelationCommand.metrics$lzycompute(InsertIntoHadoopFsRelationCommand.scala:47)

at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelationCommand.metrics(InsertIntoHadoopFsRelationCommand.scala:47)

at org.apache.spark.sql.execution.command.DataWritingCommandExec.metrics$lzycompute(commands.scala:100)

at org.apache.spark.sql.execution.command.DataWritingCommandExec.metrics(commands.scala:100)

at org.apache.spark.sql.execution.SparkPlanInfo$.fromSparkPlan(SparkPlanInfo.scala:56)

at org.apache.spark.sql.execution.SQLExecution$$anonfun$withNewExecutionId$1.apply(SQLExecution.scala:76)

at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:125)

at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:73)

at org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:676)

at org.apache.spark.sql.DataFrameWriter.saveToV1Source(DataFrameWriter.scala:285)

at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:271)

at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:229)

at org.apache.spark.sql.DataFrameWriter.parquet(DataFrameWriter.scala:566)

看起来是和SparkContext有关的问题。

final SparkSession sparkSession = SparkSession

.builder()

.appName("Java Spark SQL job")

.getOrCreate();

ds.write().mode("overwrite").parquet(path);

有什么建议或解决办法吗？谢谢

更新1：

ds的创建有点复杂，但我将尝试列出主要的调用堆栈，如下所示：

流程1：

会话.read(二)以拼花(路径)为来源；

过程2：

在步骤6之后，我使用下一个进程的相同spark会话循环回到步骤1。

我可以在进程1完成后找到日志，这似乎表明SparkContext在进程2之前已被销毁：

INFO SparkContext: Successfully stopped SparkContext