Spark 报错 Failed to delete: C:\Users\lvacz\AppData\Local\Temp\spark-*

已于 2022-04-29 12:37:58 修改
阅读量3.7k 收藏 9
点赞数 5
分类专栏： # Apache Spark 文章标签： spark win10
于 2021-05-26 13:32:17 首次发布
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/qq_39945938/article/details/117288195
版权

一、问题

在win10，local模式执行完spark任务后不论是否可以执行出结果，都会报错：

Failed to delete: C:\Users\lvacz\AppData\Local\Temp\spark-7921735f-07fa-45db-875e-5a6440eb7e79

部分日志：

21/05/26 13:01:34 WARN SparkEnv: Exception while deleting Spark temp dir: C:\Users\lvacz\AppData\Local\Temp\spark-7921735f-07fa-45db-875e-5a6440eb7e79\userFiles-af608dcd-6f5c-44a3-a295-3667472e8936
java.io.IOException: Failed to delete: C:\Users\lvacz\AppData\Local\Temp\spark-7921735f-07fa-45db-875e-5a6440eb7e79\userFiles-af608dcd-6f5c-44a3-a295-3667472e8936\org.mongodb_mongo-java-driver-3.4.2.jar
	at org.apache.spark.network.util.JavaUtils.deleteRecursivelyUsingJavaIO(JavaUtils.java:144)
	at org.apache.spark.network.util.JavaUtils.deleteRecursively(JavaUtils.java:118)
	at org.apache.spark.network.util.JavaUtils.deleteRecursivelyUsingJavaIO(JavaUtils.java:128)
	at org.apache.spark.network.util.JavaUtils.deleteRecursively(JavaUtils.java:118)
	at org.apache.spark.network.util.JavaUtils.deleteRecursively(JavaUtils.java:91)
	at org.apache.spark.util.Utils$.deleteRecursively(Utils.scala:1062)
	at org.apache.spark.SparkEnv.stop(SparkEnv.scala:103)
	at org.apache.spark.SparkContext$$anonfun$stop$11.apply$mcV$sp(SparkContext.scala:1974)
	at org.apache.spark.util.Utils$.tryLogNonFatalError(Utils.scala:1340)
	at org.apache.spark.SparkContext.stop(SparkContext.scala:1973)
	at org.apache.spark.SparkContext$$anonfun$2.apply$mcV$sp(SparkContext.scala:575)
	at org.apache.spark.util.SparkShutdownHook.run(ShutdownHookManager.scala:216)
	at org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(ShutdownHookManager.scala:188)
	at org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1$$anonfun$apply$mcV$sp$1.apply(ShutdownHookManager.scala:188)
	at org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1$$anonfun$apply$mcV$sp$1.apply(ShutdownHookManager.scala:188)
	at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1945)
	at org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1.apply$mcV$sp(ShutdownHookManager.scala:188)
	at org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1.apply(ShutdownHookManager.scala:188)
	at org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1.apply(ShutdownHookManager.scala:188)
	at scala.util.Try$.apply(Try.scala:192)
	at org.apache.spark.util.SparkShutdownHookManager.runAll(ShutdownHookManager.scala:188)
	at org.apache.spark.util.SparkShutdownHookManager$$anon$2.run(ShutdownHookManager.scala:178)
	at org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:54)
21/05/26 13:01:34 ERROR ShutdownHookManager: Exception while deleting Spark temp dir: C:\Users\lvacz\AppData\Local\Temp\spark-7921735f-07fa-45db-875e-5a6440eb7e79
java.io.IOException: Failed to delete: C:\Users\lvacz\AppData\Local\Temp\spark-7921735f-07fa-45db-875e-5a6440eb7e79\userFiles-af608dcd-6f5c-44a3-a295-3667472e8936\org.mongodb_mongo-java-driver-3.4.2.jar
	at org.apache.spark.network.util.JavaUtils.deleteRecursivelyUsingJavaIO(JavaUtils.java:144)
	at org.apache.spark.network.util.JavaUtils.deleteRecursively(JavaUtils.java:118)
	at org.apache.spark.network.util.JavaUtils.deleteRecursivelyUsingJavaIO(JavaUtils.java:128)
	at org.apache.spark.network.util.JavaUtils.deleteRecursively(JavaUtils.java:118)
	at org.apache.spark.network.util.JavaUtils.deleteRecursivelyUsingJavaIO(JavaUtils.java:128)
	at org.apache.spark.network.util.JavaUtils.deleteRecursively(JavaUtils.java:118)
	at org.apache.spark.network.util.JavaUtils.deleteRecursively(JavaUtils.java:91)
	at org.apache.spark.util.Utils$.deleteRecursively(Utils.scala:1062)
	at org.apache.spark.util.ShutdownHookManager$$anonfun$1$$anonfun$apply$mcV$sp$3.apply(ShutdownHookManager.scala:65)
	at org.apache.spark.util.ShutdownHookManager$$anonfun$1$$anonfun$apply$mcV$sp$3.apply(ShutdownHookManager.scala:62)
	at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
	at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186)
	at org.apache.spark.util.ShutdownHookManager$$anonfun$1.apply$mcV$sp(ShutdownHookManager.scala:62)
	at org.apache.spark.util.SparkShutdownHook.run(ShutdownHookManager.scala:216)
	at org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(ShutdownHookManager.scala:188)
	at org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1$$anonfun$apply$mcV$sp$1.apply(ShutdownHookManager.scala:188)
	at org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1$$anonfun$apply$mcV$sp$1.apply(ShutdownHookManager.scala:188)
	at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1945)
	at org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1.apply$mcV$sp(ShutdownHookManager.scala:188)
	at org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1.apply(ShutdownHookManager.scala:188)
	at org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1.apply(ShutdownHookManager.scala:188)
	at scala.util.Try$.apply(Try.scala:192)
	at org.apache.spark.util.SparkShutdownHookManager.runAll(ShutdownHookManager.scala:188)
	at org.apache.spark.util.SparkShutdownHookManager$$anon$2.run(ShutdownHookManager.scala:178)
	at org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:54)
21/05/26 13:01:34 ERROR ShutdownHookManager: Exception while deleting Spark temp dir: C:\Users\lvacz\AppData\Local\Temp\spark-7921735f-07fa-45db-875e-5a6440eb7e79\userFiles-af608dcd-6f5c-44a3-a295-3667472e8936
java.io.IOException: Failed to delete: C:\Users\lvacz\AppData\Local\Temp\spark-7921735f-07fa-45db-875e-5a6440eb7e79\userFiles-af608dcd-6f5c-44a3-a295-3667472e8936\org.mongodb_mongo-java-driver-3.4.2.jar
	at org.apache.spark.network.util.JavaUtils.deleteRecursivelyUsingJavaIO(JavaUtils.java:144)
	at org.apache.spark.network.util.JavaUtils.deleteRecursively(JavaUtils.java:118)
	at org.apache.spark.network.util.JavaUtils.deleteRecursivelyUsingJavaIO(JavaUtils.java:128)
	at org.apache.spark.network.util.JavaUtils.deleteRecursively(JavaUtils.java:118)
	at org.apache.spark.network.util.JavaUtils.deleteRecursively(JavaUtils.java:91)
	at org.apache.spark.util.Utils$.deleteRecursively(Utils.scala:1062)
	at org.apache.spark.util.ShutdownHookManager$$anonfun$1$$anonfun$apply$mcV$sp$3.apply(ShutdownHookManager.scala:65)
	at org.apache.spark.util.ShutdownHookManager$$anonfun$1$$anonfun$apply$mcV$sp$3.apply(ShutdownHookManager.scala:62)
	at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
	at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186)
	at org.apache.spark.util.ShutdownHookManager$$anonfun$1.apply$mcV$sp(ShutdownHookManager.scala:62)
	at org.apache.spark.util.SparkShutdownHook.run(ShutdownHookManager.scala:216)
	at org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(ShutdownHookManager.scala:188)
	at org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1$$anonfun$apply$mcV$sp$1.apply(ShutdownHookManager.scala:188)
	at org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1$$anonfun$apply$mcV$sp$1.apply(ShutdownHookManager.scala:188)
	at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1945)
	at org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1.apply$mcV$sp(ShutdownHookManager.scala:188)
	at org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1.apply(ShutdownHookManager.scala:188)
	at org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1.apply(ShutdownHookManager.scala:188)
	at scala.util.Try$.apply(Try.scala:192)
	at org.apache.spark.util.SparkShutdownHookManager.runAll(ShutdownHookManager.scala:188)
	at org.apache.spark.util.SparkShutdownHookManager$$anon$2.run(ShutdownHookManager.scala:178)
	at org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:54)

二、解决

根据自己情况自行选择解决方式

1 本地安装部署过spark

1.1 修改Spark日志配置

  1. 找到%SPARK_HOME%\conf下的log4j.properties，如果没有log4j.properties，重命名log4j.properties.template为log4j.properties
  2. 用文本编辑器编辑log4j.properties
  3. 在文件末尾追加如下2行配置
log4j.logger.org.apache.spark.util.ShutdownHookManager=OFF
log4j.logger.org.apache.spark.SparkEnv=ERROR

再次执行就不会报错了，但存在一个问题，这个零时目录(C:\Users\lvacz\AppData\Local\Temp)下spakr创建的以spark开头的临时文件会越来越多。

1.2 编写删除临时文件脚本

请注意路径里的lvacz是电脑的用户名，请替换为自己的对应路径

在%SPARK_HOME%\conf目录创建powershell脚本，脚本名字为rm_spark_tmpdir.ps1
在这里插入图片描述

Remove-Item C:\Users\lvacz\AppData\Local\Temp\spark-* -Recurse -Force

鼠标移动到脚本文件名上 ——> 右键 ——> 点创建快捷方式
重命名创建好的快捷方式（非必须）

1.3 添加到开机启动

将上面创建的（如果重命名了就是重命名后的）快捷方式移动到如下路径

%APPDATA%\Microsoft\Windows\Start Menu\Programs\Startup

2 只有项目，没有本地部署spark

  1. 在该项目的resource目录下新建名为org/apache/spark/log4j-defaults.properties的文件

在文件中添加如下内容

#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
#
#    http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#

# Set everything to be logged to the console
#log4j.rootCategory=INFO, console
log4j.rootCategory=ERROR, console
log4j.logger.org.apache.spark.util.ShutdownHookManager=OFF
log4j.appender.console=org.apache.log4j.ConsoleAppender
log4j.appender.console.target=System.err
log4j.appender.console.layout=org.apache.log4j.PatternLayout
log4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n

# Set the default spark-shell log level to WARN. When running the spark-shell, the
# log level for this class is used to overwrite the root logger's log level, so that
# the user can have different defaults for the shell and regular Spark apps.
log4j.logger.org.apache.spark.repl.Main=WARN

# Settings to quiet third party logs that are too verbose
log4j.logger.org.spark_project.jetty=WARN
log4j.logger.org.spark_project.jetty.util.component.AbstractLifeCycle=ERROR
log4j.logger.org.apache.spark.repl.SparkIMain$exprTyper=INFO
log4j.logger.org.apache.spark.repl.SparkILoop$SparkILoopInterpreter=INFO

# SPARK-9183: Settings to avoid annoying messages when looking up nonexistent UDFs in SparkSQL with Hive support
log4j.logger.org.apache.hadoop.hive.metastore.RetryingHMSHandler=FATAL
log4j.logger.org.apache.hadoop.hive.ql.exec.FunctionRegistry=ERROR

# Parquet related logging
log4j.logger.org.apache.parquet.CorruptStatistics=ERROR
log4j.logger.parquet.CorruptStatistics=ERROR

再运行该项目中的spark任务，这个时候应该就不会报错了

后续操作参考1 本地安装部署过spark部分的1.2、1.3配置即可

参考资料

https://www.jianshu.com/p/547892d6657e

终回首
关注
  • 5
    点赞
  • 踩
  • 9
    收藏
    觉得还不错? 一键收藏
  • 5
    评论
专栏目录
解决Maven打包报错：Failed to clean project: Failed to delete
JacaCao的博客
04-27 1万+
一、错误描述 在maven项目开发的过程中，会经常进行开发打包发布新的版本，执行maven的指令如：clean、install等。有时候当我们执行maven的clean指令时，控制台会报Failed to clean project: Failed to delete 错误。 1.错误日志 2.错误说明 [ERROR] Failed to execute goal org.apache...
Spark中Java.io.Exception_Failed to delete /tmp/spark-....临时文件
YangSir7的博客
04-03 2514
Spark频繁提交任务导致崩盘         1：问题描述：Java.IO.Exception                  Failedto delete /temp/spark-----删除临时文件失败         2：问题原因：                  提交spark任务太过频繁，临时文件产生太多，导致磁盘溢出；         3：问题概述：              ...
IDEA打包时clean报错Failed to delete
weixin_46483006的博客
11-06 7614
1.在开始搜索框中输入“资源监视器”----切换到CPU页面-----在关联的句柄的输入框中----输入target,回车，就能找到被占用的进程。现象：控制台报Failed to clean project:Failed to delete。原因：target文件可能时编译的文件被其他程序占用，导致资源无法回收。2.选中搜索结果，鼠标右键，结束进程。
java.io.IOException: Failed to delete: C:\Users\dell\AppData\Local\Temp\spark- in windows
Solar's Blog
10-14 5081
实验环境 windows10 spark2.4 Scala 2.11.12 问题描述 \sbtSpark1_jar>spark-submit --class com.spark.WordCount.WordCount sbtSpark。jar java.io.IOException: Failed to delete: C:\Users\dell\AppData\Local\Temp\s...
Failed to delete XXXX.jar
最新发布
zcxbd的博客
04-12 507
idea中点击maven->对应pom->lifecycle->clean时，报错。
Failed to clean project: Failed to delete
EricXiao666的博客
06-12 1272
今天执行mvn clean 命令时提示如下错误信息： 原因分析： 之前编译的工程还在运行，文件被占用导致无法删除相关文件。 解决方法： 如果是通过外部tomcat启动的项目，则停止tomcat服务。 其他方式可以通过手动关闭相关进程解决。 ...
Failed to clean project: Failed to delete F:\x\x\target\xx\xxx.jar
jdk_wangtaida的博客
09-04 2055
前言： 我们在开发maven项目的时候，会经常开发打包，执行maven的指令：clean、install等。当有的时候我们执行maven的clean指令时，控制台会报如标题的一个错误，这篇文章讲解下怎么解决这个问题。 正文： 1.我的控制台错误日志是这样的 [INFO] Scanning for projects... [INFO] ...
spark-data-repair-plugin:提供Spark中数据修复功能的统计工具包
03-11
这是一个实验性原型，可在分布式计算框架Spark上提供统计数据修复功能。 干净一致的数据可以对下游处理产生积极影响； 干净的数据使报告和机器学习更加准确，并且具有约束（例如，功能依赖性）的数据对于有效的查询...
org.mongodb.spark:mongo-spark-connector_2.11:1.1.0
11-30
mongodb-spark官方连接器，运行spark-submit --packages org.mongodb.spark:mongo-spark-connector_2.11:1.1.0可以自动下载，国内网络不容易下载成功，解压后保存到~/.ivy2目录下即可。
Spark-JAVA-Study:火花程序
05-16
在Spark-JAVA-Study项目中，我们关注的是Apache Spark与Java的结合，这是一套用于大数据处理的强大工具。Apache Spark是分布式计算领域的明星框架，它以其高性能、易用性和广泛的功能集而闻名。Spark的核心特性在于...
配套资源 github : spark-data-transmission 项目
06-13
GP_connector jar 包
maven打包出错: Failed to clean project: Failed to delete
热门推荐
u014032123的博客
10-19 1万+
maven打包出错: Failed to clean project: Failed to delete 出现这种错误，通常是由于您已启动了另一个tomcat 进程，导致报错,关闭tomcat进程即可。
解决 报错 Failed to delete XX.jar
qq_31647655的博客
04-07 649
打包时报这个错 关闭运行的tomcat就行。
问题： Failed to delete XXX\target\tomcat\logs\access_log.2020-07-19
刘亚男
07-26 2030
在点击Lifecycle上的clean之前一定要关掉当前启动的程序
Spark failed to delete temp directory
UncleTian的博客
01-17 2088
一，错误日志 17/05/02 11:56:57 INFO ShutdownHookManager: Deleting directory C:\Users\arpitbh\AppData\Local\Temp\spark-03f14dbe-1802-40ca-906c-af8de0f462f9 17/05/02 11:56:57 ERROR ShutdownHookManager: Except...
Android studio 编译项目时出现Failed to delete some children. This might happen because a process……错误处理
Bentley_li的博客
07-21 3561
Android studio 编译项目时出现Failed to delete some children. This might happen because a process……错误处理
Unable to delete directory: F:\Android\AndroidStudioWorkSpace\AIDLDemo\app\build\outputs\apk终极解决方案
zsr0526的博客
11-28 1892
Execution failed for task ':app:clean'.   > Unable to delete directory: F:\Android\AndroidStudioWorkSpace\AIDLDemo\app\build\outputs\apk   最近总是频繁的遇到这个问题，搜了好多解决方案，大多是关于怎么卸载apk文件的，或者重启怎么的。后来终于找到
python 一个.py文件如何调用另一个.py文件中的类和函数
STR_Liang的博客
08-13 9720
在同一个文件夹下 调用函数： A.py文件： def add(x,y): print('和为：%d'%(x+y)) B.py文件： import A A.add(1,2) 或 from A import add add(1,2) 调用类： A.py文件: class A: def __init__(self,xx,yy): self.x=xx self.y=yy def add(self): pr.
spark本地服务器tmp文件,【解决】Spark运行时产生的临时目录的问题
weixin_42341237的博客
08-06 2406
[Author]: kwu今天，在执行spark-sql时，查询数据量超过1.7亿行，数据量大小38G，出现系统盘占用突然变高的情况，监控如下：检查为 /tmp 目录下，spark生成的临时目录占用了大量的磁盘空间，生成的spark临时文件超过15G了。解决方案为：1、删除 /tmp/spark* 的文件rm -rf /tmp/spark*2、修改spark执行时临时目录的配置，在 conf ...
spark-shell报错：Failed to find Spark jars directory. You need to build Spark before running this program.
07-16
您好！对于这个报错信息，可能是因为您在运行 spark-shell 命令之前没有正确构建 Spark。请按照以下步骤进行操作： 1. 确保您已经正确安装了 Spark，并且设置了 SPARK_HOME 环境变量。您可以通过运行 `echo $SPARK_HOME` 来检查环境变量是否正确设置。 2. 确认您已经在正确的目录下运行了 spark-shell 命令。通常情况下，应该在 Spark 的安装目录下执行该命令。 3. 如果您已经正确设置了 SPARK_HOME 环境变量，但是仍然遇到这个问题，可能是由于缺少必要的 Spark JAR 文件。您可以尝试重新构建 Spark，或者从 Spark 的官方网站下载已编译的二进制版本。 如果您还有其他问题，请随时提问！

“相关推荐”对你有帮助么？

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论 5
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值