java执行python中文有问题_Py4JJavaError:调用

我刚认识派斯帕克。我一直在用测试样本编写代码。一旦我在更大的文件(3gb压缩)上运行代码。我的代码只是做一些过滤和连接。我一直有关于py4J的错误

任何帮助都是有用的，谢谢。from pyspark.sql import SparkSession

from pyspark.conf import SparkConf

ss = SparkSession \

.builder \

.appName("Example") \

.getOrCreate()

ss.conf.set("spark.sql.execution.arrow.enabled", 'true')

df = ss.read.csv(directory + '/' + filename, header=True, sep=",")

# Some filtering and groupbys...

df.show()

返回Py4JJavaError: An error occurred while calling o88.showString.

: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in

stage 3.0 failed 1 times, most recent failure: Lost task 0.0 in stage 3.0 (TID

1, localhost, executor driver): java.lang.OutOfMemoryError: Java heap space

...

Caused by: java.lang.OutOfMemoryError: Java heap space

更新：

我用的是py4j 10.7，刚刚更新到10.8

更新(1)：添加spark.driver.memory：ss = SparkSession \

.builder \

.appName("Example") \

.config("spark.driver.memory", "16g")\

.getOrCreate()

汇总返回错误：ERROR:py4j.java_gateway:An error occurred while trying to connect to the Java server (127.0.0.1:38004)

py4j.protocol.Py4JNetworkError: Answer from Java side is empty

"Error while receiving", e, proto.ERROR_ON_RECEIVE)

py4j.protocol.Py4JNetworkError: Error while receiving

Py4JError

Py4JError: An error occurred while calling o94.showString

半解决：这似乎是一个普遍的记忆问题。我用32g内存启动了一个2xlarge实例。程序运行时没有错误。

知道了这一点，还有什么其他的配置选项可以帮助我不必运行昂贵的实例吗？

谢谢大家。