当在pyspark中保存RDD时，如何覆盖已有的输出路径。首次保存正常，再次保存时遇到文件已存在的问题。在DataFrame中可以使用`overwrite()`来覆盖，但RDD对象上直接使用会报错。解决方案包括在Scala中设置`spark.hadoop.validateOutputSpecs`为`False`，或者转换为DataFrame后使用`SaveMode.Overwrite`。
How to overwrite
RDD output objects any existing path when we are saving time.
test1:
975078|56691|2.000|20171001_926_570_1322
975078|42993|1.690|20171001_926_570_1322
975078|46462|2.000|20171001_926_570_1322
975078|87815|1.000|20171001_926_570_1322
rdd=sc.textFile('/home/administrator/work/test1').map( lambda x: x.split("|")[:4]).map( lambda r: Row( user_code = r[0],item_code = r[1],qty = float(r[2])))
rdd.coalesce(1).saveAsPickleFile("/home/administrator/work/foobar_seq1")
The first time it is saving properly. now again I removed one line from the input
file and saving RDD same location, it show file has existed.
rdd.coalesce(1).saveAsPickleFile("/home/administrator/work/foobar_seq1")
For example, in dataframe we can overwrite existing path.
最低0.47元/天 解锁文章
1895
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
