linux setrlimit 内存,python – 如何使用`setr​​limit`来限制内存使用？ RLIMIT_AS过早死亡; RLIMIT_DATA,RLIMIT_RSS,RLIMIT_S...

我正在尝试使用setrlimit限制我在

Linux系统上的内存使用,以阻止我的进程崩溃机器(我的代码崩溃了高性能集群上的节点,因为一个错误导致内存消耗超过100 GIB).我似乎无法找到传递给setrlimit的正确资源;我认为它应该是居民,其中

cannot be limited with setrlimit,但我对居民,堆,堆栈感到困惑.在下面的代码中;如果我只取消注释RLIMIT_AS,则代码在numpy.ones(shape =(1000,1000,10),dtype =“f8”)的MemoryError失败,即使该数组应该只有80 MB.如果仅取消注释RLIMIT_DATA,RLIMIT_RSS或RLIMIT_STACK,则两个阵列都会成功分配,即使总内存使用量为2 GB,也可能是所需最大值的两倍.

我想让我的程序一旦尝试分配太多RAM就失败(无论如何).为什么没有RLIMIT_DATA,RLIMIT_RSS,RLIMIT_STACK和RLIMIT_AS做我的意思,什么是传递给setrlimit的正确资源？

$cat mwe.py

#!/usr/bin/env python3.5

import resource

import numpy

#rsrc = resource.RLIMIT_AS

#rsrc = resource.RLIMIT_DATA

#rsrc = resource.RLIMIT_RSS

#rsrc = resource.RLIMIT_STACK

soft, hard = resource.getrlimit(rsrc)

print("Limit starts as:", soft, hard)

resource.setrlimit(rsrc, (1e9, 1e9))

soft, hard = resource.getrlimit(rsrc)

print("Limit is now:", soft, hard)

print("Allocating 80 KB, should certainly work")

M1 = numpy.arange(100*100, dtype="u8")

print("Allocating 80 MB, should work")

M2 = numpy.arange(1000*1000*10, dtype="u8")

print("Allocating 2 GB, should fail")

M3 = numpy.arange(1000*1000*250, dtype="u8")

input("Still here…")

RLIMIT_AS行的输出取消注释：

$./mwe.py

Limit starts as: -1 -1

Limit is now: 1000000000 -1

Allocating 80 KB, should certainly work

Allocating 80 MB, should work

Traceback (most recent call last):

File "./mwe.py", line 22, in

M2 = numpy.arange(1000*1000*10, dtype="u8")

MemoryError

与其他任何未注释的运行时输出：

$./mwe.py

Limit starts as: -1 -1

Limit is now: 1000000000 -1

Allocating 80 KB, should certainly work

Allocating 80 MB, should work

Allocating 2 GB, should fail

Still here…

在最后一行,top报告我的流程使用的是379 GB VIRT,2.0 GB RES.

系统细节：

$uname -a

Linux host.somewhere.ac.uk 2.6.32-573.3.1.el6.x86_64 #1 SMP Mon Aug 10 09:44:54 EDT 2015 x86_64 x86_64 x86_64 GNU/Linux

$cat /etc/redhat-release

Red Hat Enterprise Linux Server release 6.7 (Santiago)

$free -h

total used free shared buffers cached

Mem: 2.0T 1.9T 37G 1.6G 3.4G 1.8T

-/+ buffers/cache: 88G 1.9T

Swap: 464G 4.8M 464G

$python3.5 --version

Python 3.5.0

$python3.5 -c "import numpy; print(numpy.__version__)"

1.11.1