py4j.protocol.Py4JJavaError: An error occurred while calling o116.saveAsTextFile.:

最新推荐文章于 2024-10-10 13:34:21 发布
最新推荐文章于 2024-10-10 13:34:21 发布
阅读量2.3k 收藏 2
点赞数
分类专栏： pyspark PyCharm 文章标签： hadoop spark Powered by 金山文档
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/yfy1907/article/details/128563550
版权

Traceback (most recent call last):

File "C:\Users\PycharmProjects\pythonProjectBasic\18PySpark\12.Test.py", line 47, in <module>

json_rdd.saveAsTextFile("test")

File "C:\Users\Workspaces\Env\pythonProjectBasic\lib\site-packages\pyspark\rdd.py", line 2205, in saveAsTextFile

keyed._jrdd.map(self.ctx._jvm.BytesToString()).saveAsTextFile(path)

File "C:\Users\Workspaces\Env\pythonProjectBasic\lib\site-packages\py4j\java_gateway.py", line 1322, in __call__

return_value = get_return_value(

File "C:\Users\Workspaces\Env\pythonProjectBasic\lib\site-packages\py4j\protocol.py", line 326, in get_return_value

raise Py4JJavaError(

py4j.protocol.Py4JJavaError: An error occurred while calling o116.saveAsTextFile.

: ExitCodeException exitCode=-1073741515:

at org.apache.hadoop.util.Shell.runCommand(Shell.java:1007)

at org.apache.hadoop.util.Shell.run(Shell.java:900)

at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1212)

at org.apache.hadoop.util.Shell.execCommand(Shell.java:1306)

at org.apache.hadoop.util.Shell.execCommand(Shell.java:1288)

at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:978)

at org.apache.hadoop.fs.RawLocalFileSystem.mkOneDirWithMode(RawLocalFileSystem.java:660)

at org.apache.hadoop.fs.RawLocalFileSystem.mkdirsWithOptionalPermission(RawLocalFileSystem.java:700)

at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:672)

at org.apache.hadoop.fs.RawLocalFileSystem.mkdirsWithOptionalPermission(RawLocalFileSystem.java:699)

at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:672)

at org.apache.hadoop.fs.RawLocalFileSystem.mkdirsWithOptionalPermission(RawLocalFileSystem.java:699)

at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:672)

at org.apache.hadoop.fs.ChecksumFileSystem.mkdirs(ChecksumFileSystem.java:788)

at org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.setupJob(FileOutputCommitter.java:356)

at org.apache.hadoop.mapred.FileOutputCommitter.setupJob(FileOutputCommitter.java:131)

at org.apache.hadoop.mapred.OutputCommitter.setupJob(OutputCommitter.java:265)

at org.apache.spark.internal.io.HadoopMapReduceCommitProtocol.setupJob(HadoopMapReduceCommitProtocol.scala:188)

at org.apache.spark.internal.io.SparkHadoopWriter$.write(SparkHadoopWriter.scala:79)

at org.apache.spark.rdd.PairRDDFunctions.$anonfun$saveAsHadoopDataset$1(PairRDDFunctions.scala:1091)

at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)

at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)

at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)

at org.apache.spark.rdd.RDD.withScope(RDD.scala:406)

at org.apache.spark.rdd.PairRDDFunctions.saveAsHadoopDataset(PairRDDFunctions.scala:1089)

at org.apache.spark.rdd.PairRDDFunctions.$anonfun$saveAsHadoopFile$4(PairRDDFunctions.scala:1062)

at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)

at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)

at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)

at org.apache.spark.rdd.RDD.withScope(RDD.scala:406)

at org.apache.spark.rdd.PairRDDFunctions.saveAsHadoopFile(PairRDDFunctions.scala:1027)

at org.apache.spark.rdd.PairRDDFunctions.$anonfun$saveAsHadoopFile$3(PairRDDFunctions.scala:1009)

at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)

at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)

at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)

at org.apache.spark.rdd.RDD.withScope(RDD.scala:406)

at org.apache.spark.rdd.PairRDDFunctions.saveAsHadoopFile(PairRDDFunctions.scala:1008)

at org.apache.spark.rdd.PairRDDFunctions.$anonfun$saveAsHadoopFile$2(PairRDDFunctions.scala:965)

at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)

at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)

at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)

at org.apache.spark.rdd.RDD.withScope(RDD.scala:406)

at org.apache.spark.rdd.PairRDDFunctions.saveAsHadoopFile(PairRDDFunctions.scala:963)

at org.apache.spark.rdd.RDD.$anonfun$saveAsTextFile$2(RDD.scala:1599)

at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)

at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)

at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)

at org.apache.spark.rdd.RDD.withScope(RDD.scala:406)

at org.apache.spark.rdd.RDD.saveAsTextFile(RDD.scala:1599)

at org.apache.spark.rdd.RDD.$anonfun$saveAsTextFile$1(RDD.scala:1585)

at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)

at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)

at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)

at org.apache.spark.rdd.RDD.withScope(RDD.scala:406)

at org.apache.spark.rdd.RDD.saveAsTextFile(RDD.scala:1585)

at org.apache.spark.api.java.JavaRDDLike.saveAsTextFile(JavaRDDLike.scala:564)

at org.apache.spark.api.java.JavaRDDLike.saveAsTextFile$(JavaRDDLike.scala:563)

at org.apache.spark.api.java.AbstractJavaRDDLike.saveAsTextFile(JavaRDDLike.scala:45)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:498)

at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)

at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)

at py4j.Gateway.invoke(Gateway.java:282)

at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)

at py4j.commands.CallCommand.execute(CallCommand.java:79)

at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182)

at py4j.ClientServerConnection.run(ClientServerConnection.java:106)

at java.lang.Thread.run(Thread.java:748)

原因：操作系统缺少 dll文件

解决办法：安装常用C++库合集（x86 + 64位）

百度盘： https://pan.baidu.com/s/1dsL0V9dGUYXgEvFZOYLTow

提取码：0cw9

或本站搜索：MSVBCRT_AIO_2018.07.31_X86+X64.exe 下载后安装

经尝试，放置dll:

32位\msvcr120.dll 粘贴到 C:\Windows\System32目录中

64位\msvcr120.dll 粘贴到 C:\Windows\SysWOW64目录中

这种方法没有效果，安装常用C++库合集（x86 + 64位）后解决，

特此记录

参考：https://blog.csdn.net/u013303361/article/details/88853684

Juskey iii
关注
专栏目录
An internal error occurred during: java.lang.NullPointerException
07-27
在使用MyEclipse进行Java开发的过程中，可能会遇到一个常见的问题：“An internal error occurred during: 'Calculating completion proposals...'”。这个错误通常伴随着`java.lang.NullPointerException`异常，...
spark读取文件 报错：py4j.protocol.Py4JJavaError
qq_39954916的博客
08-28 1178
# read data from hdfs and transform RDD[Row] page_views_rdd = sc\ .textFile("/user/hive/warehouse/page_views")\ .map(map_func) # Create DataFrame page_views_df = sqlContext.createDataFrame(page_views_rdd) 原代码如上，要读取的文件保存...
spark 报错 py4j.protocol.Py4JJavaError错误
从零开始学习python --zeropython
09-04 1万+
Python java:0 >>> r = sc.textFile("~/Desktop/heimao.md") >>> r.take(1) Traceback (most recent call last): File &#...
记录VASP计算报错及解决办法
最新发布
weixin_69031174的博客
10-10 312
解决办法：将LREAL ＝.TRUE.换成LREAL ＝ Auto。
【Pyspark】报错：py4j.protocol.Py4JJavaError
sunflower_sara的机器学习园地
10-30 4174
py4j.protocol.Py4JJavaError 找到报错信息： 310 raise Py4JJavaError( 311 "An error occurred while calling {0}{1}{2}.\n". --> 312 format(tar...
py4j.protocol.Py4JJavaError: An error occurred while calling z:org.apache.spark.api.python.PythonRDD
热门推荐
u011250186的博客
01-05 1万+
今天小编用Python编写Spark程序报了如下异常： py4j.protocol.Py4JJavaError: An error occurred while calling z:org.apache.spark.api.python.PythonRDD.collectAndServe. : java.lang.IllegalArgumentException: Unsupported class file major version 55 从网上找到的解决方案是JDK版本问题，于是乎小编将Ja...
py4j.protocol.Py4JJavaError: An error occurred while calling o32.jdbc.
Zsl0515的博客
05-23 1908
然后将其中的jar包mysql-connector-java-8.0.33.jar放入spark的jars目录下，例如/usr/local/spark/jars。后来发现没有这个驱动，这里注意：要下载和mysql数据库版本一样的驱动。在使用pyspark连接mysql数据库时出现了这个错误。连接数据库时可以发现mysql数据库的版本。然后在去下载对应版本的mysql的jar包。然后重新启动pyspark。
Windows系统运行pyspark报错：Py4JJavaError
wzy_xd666的博客
11-02 3746
import findspark findspark.init() #因为转载时间过长找不到spark所以报错
关于eclipse安装spring插件报错An error occurred while collecting items to be installed...解决方案
08-25
主要介绍了关于eclipse安装spring插件报错An error occurred while collecting items to be installed...解决方案，文中通过示例代码介绍的非常详细，对大家的学习或者工作具有一定的参考学习价值，需要的朋友们下面...
An internal error occurred during: \"Building workspace\". Java heap space
04-11
标题 "An internal error occurred during: \"Building workspace\". Java heap space" 描述的是一个常见的Eclipse集成开发环境（IDE）中的问题，当用户尝试构建工作区时遇到内存不足的情况。这个错误表明Eclipse在...
解决 github项目clone报错 Failed connect to github.com:443; Connection
01-09
Error while executing: npm ERR! /usr/bin/git ls-remote -h -t https://github.com/nhn/raphael.git npm ERR! npm ERR! fatal: unable to access 'https://github.com/nhn/raphael.git/': Failed connect to ...
log4j-1.2.8.jar包，在项目中直接导入即可。
03-11
logger.error("An error occurred", new Exception("Sample exception")); } } ``` 在这个例子中，我们首先获取了与`Main`类相关的Logger，然后设置其日志级别为DEBUG，接着记录了不同级别的日志信息。 7. **性能...
PySpark的运行出错：Py4JJavaError【python为3.9==＞pyspark版本为3.0】
u013250861的博客
08-02 1891
对策Python的安装目录里面复制python.exe，并改名为python3.exe。
pyspark使用报错记录
Mugo_Moon的博客
10-09 2296
1、启动spark时，提示JAVA_HOME not set （1）下载jdk-8u291-linux-x64.tar.gz （2）解压到/usr/local/java目录下 （3）在~/.bashrc中添加以下内容 export JAVA_HOME="/usr/local/java/jdk1.8.0_291" export PATH=$JAVA_HOME/bin:$PATH （4）source ~/.bashrc （5）测试 (py3_spark) [root@100-020-gpus
py4j.protocol.Py4JJavaError: An error occurred while calling o45.load.
sinat_32618807的博客
07-13 1万+
py4j.protocol.Py4JJavaError: An error occurred while calling o45.load. 安装对应MySQL版本的Connector J 8.0（默认地址为C:\Program Files(x86)\MySQL\Connector J 8.0)，再把这个文件夹下的mysql-connector-java-8.0.16.jar复制到spark安装目...
py4j.protocol.Py4JJavaError: An error occurred while calling o35.load.错误
qq_46076785的博客
09-13 9521
py4j.protocol.Py4JJavaError: An error occurred while calling o35.load.错误 pyspark运行报错，降低jdk版本并没有什么用 报错代码 代码如下（示例）： 解决办法 ： C:\Program Files\Java\jdk1.8.0_141\jre\lib\ext路径下缺失mysql-connector-java-8.0.16.jar包，下载一个jar包放到当前路径下 最后关闭pycharm，重新打开再次运行，问题得以.
py4j.protocol.Py4JJavaError: An error occurred while calling None.org.apache.spark.api.java.JavaSpar
rong_king_的博客
09-18 3241
py4j.protocol.Py4JJavaError: An error occurred while calling None.org.apache.spark.api.java.JavaSparkContext. : java.io.FileNotFoundException: File file:/Users/didi/Desktop/IdeaProjects/spark_learn/log does not exist at org.apache.hadoop.fs.RawLocalFileSy
pysaprk报错：Py4JJavaError
wanghouyushengbu的博客
05-30 1万+
pysaprk：toPandas报错Py4JJavaError Traceback (most recent call last)Py4JJavaError: An error occurred while calling o946.collectToPython.原因：Java版本问题，现在使用的版本过高，切换到1.8问题解决。ps：pyt...
py4j.protocol.Py4JJavaError: An error occurred while calling o67.jdbc.
07-28
- *3* [py4j.protocol.Py4JJavaError: An error occurred while calling o116.saveAsTextFile.:](https://blog.csdn.net/yfy1907/article/details/128563550)[target="_blank" data-report-click={"spm":"1018.2226....
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值