executor.Executor: Managed memory leak detected; size = 37247642 bytes, TID = 5

最新推荐文章于 2021-10-26 14:24:23 发布

afei2530

最新推荐文章于 2021-10-26 14:24:23 发布

阅读量469

点赞数

文章标签：大数据 java

原文链接：http://www.cnblogs.com/rocky-AGE-24/p/7512071.html

版权

https://stackoverflow.com/questions/34359211/debugging-managed-memory-leak-detected-in-spark-1-6-0

https://stackoverflow.com/questions/33518992/spark-executor-managed-memory-leak-detected

I saw this exception while running spark streaming on EMR 16/04/14 13:49:10 WARN memory.TaskMemoryManager: leak 32.3 MB memory from org.apache.spark.unsafe.map.BytesToBytesMap@34158d5f 16/04/14 13:49:10 ERROR executor.Executor: Managed memory leak detected; size = 33816576 bytes, TID = 2942915 16/04/14 13:49:10 ERROR executor.Executor: Exception in task 22.0 in stage 35684.0 (TID 2942915) java.lang.OutOfMemoryError: Unable to acquire 262144 bytes of memory, got 220032 – Nipun Apr 20 '16 at 9:27

Were you able to resolve it? I am facing a similar issue of memory leak in spark 1.6.2 – Anchika Agarwal Mar 24 at 18:43

I think in my case it was SPARK-14560. It is fixed in Spark 2.0.0 and we are using 2.1.0 now, so all is good. – Daniel Darabos Mar 25 at 20:10

does all these comment mean, if I can not upgrade to the 'good' version yet due to company IT readiness, I can safely ignore the error? I am currently using spark 1.6.0 and having errors like this when some dataframe does 'distinct' and then 'head'. – Minnie Shi Aug 8 at 9:55

Well, if ignoring it is your only option, I suggest you ignore it :). It may lead to out-of-memory issues, being a warning about memory mis-accounting. – Daniel Darabos Aug 8 at 10:31

转载于:https://www.cnblogs.com/rocky-AGE-24/p/7512071.html