我在一台机器上运行spark,24核,48G内存。在
我能够训练一个拥有600万用户、120万个项目、2.16亿个操作(浏览/购买)的隐式模型
现在,我正在尝试运行700万用户和150万个项目的建议,以及4.4亿个用户对项目的操作。在
我使用20个执行器,驱动程序内存15G,执行器内存4G
8级训练,15次迭代。在
我在使用ALS.训练隐式. 在model = ALS.trainImplicit(training_RDD, rank, seed=seed, iterations=iterations, lambda_=regularization_parameter, alpha=config.alpha)
File "/usr/local/spark/python/lib/pyspark.zip/pyspark/mllib/recommendation.py", line 314, in trainImplicit
File "/usr/local/spark/python/lib/pyspark.zip/pyspark/mllib/common.py", line 130, in callMLlibFunc
File "/usr/local/spark/python/lib/pyspark.zip/pyspark/mllib/common.py", line 123, in callJavaFunc
File "/usr/local/spark/python/lib/py4j-0.10.4-src.zip/py4j/java_gateway.py", line 1133, in __call__
File "/usr/local/spark/python/lib/pyspark.zip/pyspark/sql/utils.py", line 63, in deco
File "/usr/local/spark/python/lib/py4j-0.10.4-src.zip/py4j/protocol.py", line 319, in get_return_value
py4j.protocol.Py4JJavaError: An error occurred while calling o86.trainImplicitALSModel.
: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 4.0 failed 1 times, most recent failure: Lost task 0.0 in stage 4.0 (TID 44, localhost, executor driver): java.lang.OutOfMemoryError: Java heap space
at scala.collection.mutable.ArrayBuilder$ofInt.mkArray(ArrayBuilder.scala:323)
我不知道如何纠正这个错误。
来自错误:阶段4.0中的任务0.0丢失(TID 44,本地主机,执行器驱动程序):java.lang.OutOfMemoryError:Java堆空间,我可以知道执行器内存不足。在
我试过增加执行器内存,减少驱动程序内存,但是没有用,我仍然得到同样的错误。在