pyspark 错误 IndexError: pop from an empty deque

问题

在调试wide and deep 网络，对数据做预处理时，对pages_views进行sql查询时出现“IndexError: pop from an empty deque”错误。

代码

train_set_df = spark.sql('''
SELECT * FROM clicks_train_joined t
WHERE NOT EXISTS (SELECT display_id FROM validation_display_ids
WHERE display_id = t.display_id)''').cache()
print("train_set_df.count() =", train_set_df.count())

log

ERROR:py4j.java_gateway:An error occurred while trying to connect to the Java server (127.0.0.1:41935)
Traceback (most recent call last):
  File "/usr/local/spark/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 929, in _get_connection
    connection = self.deque.pop()
IndexError: pop from an empty deque

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/spark/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1067, in start
    self.socket.connect((self.address, self.port))
ConnectionRefusedError: [Errno 111] Connection refused

原因

网上搜索了一番，中文页面较少相关信息，在github上找到一个原因，测试了一下，解决了我的问题，因为在cache rdd_1145_1中没有足够的空间了。是JVM中memory溢出问题。

解决方法

将driver-memory 设小，设成4G

conf = SparkConf().setMaster('local[*]').set('spark.executor.memory', '4g').set('spark.driver.memory', '20g').set(
    "spark.local.dir", SPARK_TEMP_FOLDER)

参考链接：
https://github.com/bartdag/py4j/issues/325