c python 内存冲突_python内存不足错误的解决方法是什么？

I am reading a x,y,z point file (LAS) into python and have run into memory errors. I am interpolating unknown points between known points for a project I am working on. I began working with small files (< 5,000,000 points) and was able to read/write to a numpy array and python lists with no problem. I have received more data to work with (> 50,000,000 points) and now my code fails with a MemoryError.

What are some options for handling such large amounts of data? I do not have to load all data into memory at once, but I will need to look at neighboring points using scipy kd-tree I am using Python 2.7 32 bit on a 64 bit Windows XP OS.

Thanks in advance.

EDIT: Code is posted below. I took out code for long calculations and variable definitions.

from liblas import file

import numpy as np

f = file.File(las_file, mode='r')

num_points = int(f.__len__())

dt = [('x', 'f4'), ('y', 'f4'), ('z', 'f4'), ('i', 'u2'), ('c', 'u1'), ('t', 'datetime64[us]')]

xyzict = np.empty(shape=(num_points,), dtype = dt)

counter = 0

for p in f:

newrow = (p.x, p.y, p.z, p.intensity, p.classification, p.time)

xyzict[counter] = newrow

counter += 1

dropoutList = []

counter = 0

for i in np.nditer(xyzict):

# code to define P1x, P1y, P1z, P1t

if counter != 0:

# code to calculate n, tDiff, and seconds

if n > 1 and n < scanN:

# code to find v and vD

for d in range(1, int(n-1)):

# Code to interpolate x, y, z for points between P0 and P1

# Append tuple of x, y, and z to dropoutList

dropoutList.append(vD)

# code to set x, y, z, t for next iteration

counter += 1

解决方案

Regardless of the amount of RAM in your system, if you are running 32-bit python, you will have a practical limit of about 2 GB of RAM for your application. There are a number of other questions on SO that address this (e.g., see here). Since the structure you are using in your ndarray is 23 bytes and you are reading over 50,000,000 points, that already puts you at about 1 GB. You haven't included the rest of your code so it isn't clear how much additional memory is being consumed by other parts of your program.

If you have well over 2 GB of RAM in your system and you will continue to work on large data sets, you should install 64-bit python to get around this ~ 2 GB limit.