问题
在调试wide and deep 网络,对数据做预处理时,对pages_views进行sql查询时出现“IndexError: pop from an empty deque”错误。
代码
train_set_df = spark.sql('''
SELECT * FROM clicks_train_joined t
WHERE NOT EXISTS (SELECT display_id FROM validation_display_ids
WHERE display_id = t.display_id)''').cache()
print("train_set_df.count() =", train_set_df.count())
log
ERROR:py4j.java_gateway:An error occurred while trying to connect to the Java server (127.0.0.1:41935)
Traceback (most recent call last):
File "/usr/local/spark/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 929, in _get_connection
connection = self.deque.pop()
IndexError: pop from an empty deque
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/spark/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1067, in start
self.socket.connect((self.address, self.port))
ConnectionRefusedError: [Errno 111] Connection refused
原因
网上搜索了一番,中文页面较少相关信息,在github上找到一个原因,测试了一下,解决了我的问题,因为在cache rdd_1145_1中没有足够的空间了。是JVM中memory溢出问题。
解决方法
将driver-memory 设小,设成4G
conf = SparkConf().setMaster('local[*]').set('spark.executor.memory', '4g').set('spark.driver.memory', '20g').set(
"spark.local.dir", SPARK_TEMP_FOLDER)