I'm working on a project that involves using python to read, process and write files that are sometimes as large as a few hundred megabytes. The program fails occasionally when I try to process some particularly large files. It does not say 'memory error', but I suspect that is the problem (in fact it gives no reason at all for failing').
I've been testing the code on smaller files and watching 'top' to see what memory usage is like, and it typically reaches 60%. top says that I have 4050352k total memory, so 3.8Gb.
Meanwhile I'm trying to track memory usage within python itself (see my question from yesterday) with the following little bit of code:
mem = 0
for variable in dir():
variable_ = vars()[variable]
try:
if str(type(variable_))[7:12] == 'numpy':
numpy_ = True
else:
numpy_ = False
except:
numpy_ = False
if numpy_:
mem_ = variable_.nbytes
else:
mem_ = sys.getsizeof(variable)
mem += mem_
print variable+ type: '+str(type(variable_))+' size: '+str(mem_)
print 'Tot