pyspark写入mysql内存溢出_python - 如何处理这个PySpark问题？ - 堆栈内存溢出

最新推荐文章于 2024-06-11 23:44:57 发布

1X15A126

最新推荐文章于 2024-06-11 23:44:57 发布

阅读量467

点赞数

文章标签： pyspark写入mysql内存溢出

本文链接：https://blog.csdn.net/weixin_33595018/article/details/113960571

版权

在尝试对0大小的rdd进行操作时，遇到内存堆映射错误，例如在`isEmpty()`或`count()`上。错误包括double free或corruption (!prev)，涉及numpy、hdbscan库和Python环境的多个组件。尝试在PySpark作业中解决内存溢出问题。

摘要由CSDN通过智能技术生成

每次尝试在大小为0的rdd上调用简单操作时，都会遇到以下问题。每次尝试在rdd中运行操作时，都会不断出现此内存堆映射错误：例如：rdd.isEmpty ()或rdd.count()

*** Error in `/home/@@@@/miniconda3/envs/Project_3/bin/python': double free or corruption (!prev): 0x00005579db3108a0 ***

======= Backtrace: =========

/lib/x86_64-linux-gnu/libc.so.6(+0x777e5)[0x7fcfce2067e5]

/lib/x86_64-linux-gnu/libc.so.6(+0x8037a)[0x7fcfce20f37a]

/lib/x86_64-linux-gnu/libc.so.6(cfree+0x4c)[0x7fcfce21353c]

/home/@@@@/.local/lib/python3.6/site-packages/numpy/core/_multiarray_umath.cpython-36m-x86_64-linux-gnu.so(+0x2902c)[0x7fcfcba6102c]

/home/@@@@/.local/lib/python3.6/site-packages/numpy/core/_multiarray_umath.cpython-36m-x86_64-linux-gnu.so(+0x2c787)[0x7fcfcba64787]

/home/@@@@/miniconda3/envs/Project_3/lib/python3.6/site-packages/hdbscan/_hdbscan_tree.cpython-36m-x86_64-linux-gnu.so(+0x24f40)[0x7fcfbcc5ff40]

/home/@@@@/miniconda3/envs/Project_3/lib/python3.6/site-packages/hdbscan/_hdbscan_tree.cpython-36m-x86_64-linux-gnu.so(+0x2723f)[0x7fcfbcc6223f]

/home/@@@@/miniconda3/envs/Project_3/bin/python(_PyCFunction_FastCallDict+0x115)[0x5579d8873395]

/home/@@@@/miniconda3/envs/Project_3/bin/python(+0x19cc1c)[0x5579d88fcc1c]

...............

7fcfce981000-7fcfce982000 rw-p 00004000 08:02 6319360 /home/@@@@/miniconda3/envs/Project_3/lib/python3.6/lib-dynload/grp.cpython-36m-x86_64-linux-gnu.so

7fcfce982000-7fcfce984000 r--p 00000000 08:02 3808611 /home/@@@@/miniconda3/envs/Project_3/lib/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so

7fcfce984000-7fcfce988000 r-xp 00002000 08:02 3808611 /home/@@@@/miniconda3/envs/Project_3/lib/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so

7fcfce988000-7fcfce989000 r--p 00006000 08:02 3808611 /home/@@@@/miniconda3/envs/Project_3/lib/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so

7fcfce989000-7fcfce98a000 ---p 00007000 08:02 3808611 /home/@@@@/miniconda3/envs/Project_3/lib/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so

7fcfce98a000-7fcfce98b000 r--p 00007000 08:02 3808611 /home/@@@@/miniconda3/envs/Project_3/lib/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so

7fcfce98b000-7fcfce98d000 rw-p 00008000 08:02 3808611 /home/@@@@/miniconda3/envs/Project_3/lib/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so

7fcfce98d000-7fcfce98e000 r--p 00000000 08:02 6319372 /home/@@@@/miniconda3/envs/Project_3/lib/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so

7fcfce98e000-7fcfce990000 r-xp 00001000 08:02 6319372 /home/@@@@/miniconda3/envs/Project_3/lib/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so

7fcfce990000-7fcfce991000 r--p 00003000 08:02 6319372 /home/@@@@/miniconda3/envs/Project_3/lib/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so

7fcfce991000-7fcfce992000 r--p 00003000 08:02 6319372 /home/@@@@/miniconda3/envs/Project_3/lib/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so

7fcfce992000-7fcfce994000 rw-p 00004000 08:02 6319372 /home/@@@@/miniconda3/envs/Project_3/lib/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so

7fcfce994000-7fcfce99b000 r--s 00000000 08:02 5250159 /usr/lib/x86_64-linux-gnu/gconv/gconv-modules.cache

7fcfce99b000-7fcfce99c000 r--p 00025000 08:02 6035518 /lib/x86_64-linux-gnu/ld-2.23.so

7fcfce99c000-7fcfce99d000 rw-p 00026000 08:02 6035518 /lib/x86_64-linux-gnu/ld-2.23.so

7fcfce99d000-7fcfce99e000 rw-p 00000000 00:00 0

7ffcc8e26000-7ffcc8e48000 rw-p 00000000 00:00 0 [stack]

7ffcc8f86000-7ffcc8f89000 r--p 00000000 00:00 0 [vvar]

7ffcc8f89000-7ffcc8f8b000 r-xp 00000000 00:00 0 [vdso]

ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall]

[Stage 54:> (0 + 1) / 1]2019-03-01 16:16:06 ERROR Executor:91 - Exception in task 0.0 in stage 54.0 (TID 302)

org.apache.spark.SparkException: Python worker exited unexpectedly (crashed)

at org.apache.spark.api.python.BasePythonRunner$ReaderIterator$$anonfun$3.applyOrElse(PythonRunner.scala:486)

at org.apache.spark.api.python.BasePythonRunner$ReaderIterator$$anonfun$3.applyOrElse(PythonRunner.scala:475)

at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)

at org.apache.spark.api.python.PythonRunner$$anon$1.read(PythonRunner.scala:593)

at org.apache.spark.api.python.PythonRunner$$anon$1.read(PythonRunner.scala:571)

at org.apache.spark.api.python.BasePythonRunner$ReaderIterator.hasNext(PythonRunner.scala:406)

at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:37)

at scala.collection.Iterator$class.foreach(Iterator.scala:891)

at org.apache.spark.InterruptibleIterator.foreach(InterruptibleIterator.scala:28)

....

at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2101)

at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)

at org.apache.spark.scheduler.Task.run(Task.scala:121)

at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:402)

at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)

at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:408)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

at java.lang.Thread.run(Thread.java:748)

Caused by: java.io.EOFException

at java.io.DataInputStream.readInt(DataInputStream.java:392)

at org.apache.spark.api.python.PythonRunner$$anon$1.read(PythonRunner.scala:578)

... 26 more

2019-03-01 16:16:06 WARN TaskSetManager:66 - Lost task 0.0 in stage 54.0 (TID 302, localhost, executor driver): org.apache.spark.SparkException: Python worker exited unexpectedly (crashed)