Linux怎么消耗虚拟内存,为什么linux下多线程程序如此消耗虚拟内存

最近游戏已上线运营，进行

1.探索

一开始首先排除掉内存泄露，不可能每次都泄露64M内存这么巧合，为了证明我的观点，首先，我使用了valgrind。

1: valgrind --leak-check=full --track-fds=yes --log-file=./AuthServer.vlog &

然后启动

在多次使用valgrind无果以后，我开始怀疑程序内部是不是用到m

1: strace -f -e"brk,mmap,munmap" -p $(pidof AuthServer)

其结果如下：

[pid 19343] mmap(NULL, 134217728, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x7f53c8ca9000

[pid 19343] munmap(0x7f53c8ca9000, 53833728) = 0

[pid 19343] munmap(0x7f53d0000000, 13275136) = 0

[pid 19343] mmap(NULL, 8392704, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_STACK, -1, 0) = 0x7f53d04a8000

Process 19495 attached

我检查了一下trace文件也没有发现大量内存mmap动作，即便是brk动作引起的内存增长也不大。于是感觉人生都没有方向了，然后怀疑是不是文件缓存把虚拟内存占掉了，

2.灵光一现

后来，我开始减少thread的数量开始测试，在测试的时候偶然发现一个很奇怪的现象。那就是如果

#include

#include

#include

#include

using namespace std;

bool s

void * thread_run( void * )

{

while ( 1 )

{

if ( start )

{

cout << "Thread malloc" << endl;

char *buf = new char [1024];

start = 0;

}

sleep( 1 );

}

}

int main()

{

pthread_t th;

getchar ();

getchar ();

pthread_create( &th, 0, thread_run, 0 );

while ( ( getchar () ) )

{

start = 1;

}

return (0);

}

其运行结果如下图，刚开始时，进程占用虚拟内存14M，输入0，创建子线程，进程内存达到23M，这增加的10M是线程堆栈的大小(查看和设置线程堆栈大小可用ulimit –s)，第一次输入1，程序分配1k内存，整个进程增加64M虚拟内存，之后再输入2，3，各再次分配1k，内存均不再变化。

这个结果让我欣喜若狂，由于以前学习过

请注意65404这一行，种种迹象表明，这个再加上它上面那一行(在这里是132)就是增加的那个64M)。后来增加thread的数量，就会有新增thread数量相应的65404的内存块。

3.刨根问底

经过一番google和代码查看。终于知道了原来是glibc的malloc在这里捣鬼。glibc 版本大于2.11的都会有这个问题：在redhat 的官方文档上：

Red Hat Enterprise Linux 6 features version 2.11 of glibc, providing many features and enhancements, including… An enhanced dynamic memory alloThis is achieved by assigning threads their own memory pools and by avoiding locking in some situations . The amount of additional memory used for the memory pools (if any) can be controlled using the environment variables MALLOC_ARENA_TEST and MALLOC_ARENA_MAX. MALLOC_ARENA_TEST specifies that a test for the number of cores is performed once the number of memory pools reaches this

The developer, Ulrich Drepper, has a much deeper explanation on his blog: http://udrepper.livejournal.com/20948.html

Before, malloc tried to emulate a per-core memory pool. Every time when contention for all existing memory pools was detected a new pool is created. Threads stay with the last used pool if possible… This never worked 100% because a thread can be descheduled while executing a malloc call. When some other thread tries to use the memory pool used in the call it would detect contention. A second problem is that if mult

The changes which are in glibc now create per-thread memory pools.This can eli

While these changes might increase the number of memory pools which are created (and thus increase the address space they use) the number can be controlled. Because using the old mechanism there could be a new pool being created whenever there are collisions the total number could in theory be higher. Unlikely but true, so the new mechanism is more predic

… Memory use is not that much of a premium anymore and most of the memory pool doesn’t actually require memory until it is used, only address space… We have done internally some measurements of the effects of the new implementation and they can be quite dramatic.

New versions of glibc present in RHEL6 include a new  arena allocator design . In several clusters we’ve seen this new allocator cause huge amounts of virtual memory to be used, since when multiple threads perform allocations, they each get their own memory arena.  On a 64-bit system, these arenas are 64M mappings, and the maximum number of arenas is 8 times the number of cores . We’ve observed a DN process using 14GB of vmem for only 300M of resident set. This causes all kinds of nasty issues for obvious reasons.

Setting MALLOC_ARENA_MAX to a low number will restrict the number of memory arenas and bound the virtual memory, with no noticeable downside in performance – we’ve been recommending MALLOC_ARENA_MAX=4. We should set this in hadoop-env.sh to avoid this issue as RHEL6 becomes more and more common.

hadoop推荐把这个值设置为4。当然了，既然是多核的机器，而arena的引进是为了解决多线程内存分配竞争的问题，那么设置为cpu核的数量估计也是一个不错的选择。设置这个值以后最好能对你的程序做一下

mallopt(M_ARENA_MAX, xxx)如果你打算在程序代码中来设置这个东西，那么可以调用mallopt(M_ARENA_MAX, xxx)来实现，由于我们AuthServer采用了预分配的方式，在各个线程内并没有分配内存，所以不需要这种优化，在初始化的时候采用mallopt(M_ARENA_MAX, 1)将其关掉，设置为0，表示系统按CPU进行自动设置。

4.意外发现

想到tcmalloc小对象才从线程自己的内存池分配，大内存仍然从中央分配区分配，不知道glibc是如何设计的，于是将上面程序中线程每次分配的内存从1k调整为1M，果然不出所料，再分配完64M后，仍然每次都会增加1M，由此可见，新版 glibc完全借鉴了tcmalloc的思想。

忙了几天的问题终于解决了，心情大好，通过今天的问题让我知道，作为一个服务器

转自 http://blog.jobbole.com/83878/