python内存分配失败,Python MemoryError：无法分配数组内存

最新推荐文章于 2022-12-02 17:34:33 发布

weixin_39987211

最新推荐文章于 2022-12-02 17:34:33 发布

阅读量467

点赞数

文章标签： python内存分配失败

博主在尝试用numpy读取一个250MB、包含7000行和9000列的CSV文件时遇到了内存错误。尝试了多种方法如逐行读取、np.fromstring等，但都在大约512MB内存使用后失败。通过测试发现，不是总内存限制，而是多次内存分配的问题。最终，通过创建空数组并逐行填充解决了问题。

摘要由CSDN通过智能技术生成

I've got a 250 MB CSV file I need to read with ~7000 rows and ~9000 columns. Each row represents an image, and each column is a pixel (greyscale value 0-255)

I started with a simple np.loadtxt("data/training_nohead.csv",delimiter=",") but this gave me a memory error. I thought this was strange since I'm running 64-bit Python with 8 gigs of memory installed and it died after using only around 512 MB.

I've since tried SEVERAL other tactics, including:

import fileinput and read one line at a time, appending them to an array

np.fromstring after reading in the entire file

np.genfromtext

Manual parsing of the file (since all data is integers, this was fairly easy to code)

Every method gave me the same result. MemoryError around 512 MB. Wondering if there was something special about 512MB, I created a simple test progra