linux mem overcommit

 linux overcommit 机制会欺骗进程，承诺分配给他的内存空间，只有当这些page被真正touch到的时候才通过缺页机制拿到真正的物理内存。

以下摘自kernel doc

The Linux kernel supports the following overcommit handling modes

0	-	Heuristic overcommit handling. Obvious overcommits of
		address space are refused. Used for a typical system. It
		ensures a seriously wild allocation fails while allowing
		overcommit to reduce swap usage.  root is allowed to 
		allocate slightly more memory in this mode. This is the 
		default.

1	-	Always overcommit. Appropriate for some scientific
		applications. Classic example is code using sparse arrays
		and just relying on the virtual memory consisting almost
		entirely of zero pages.

2	-	Don't overcommit. The total address space commit
		for the system is not permitted to exceed swap + a
		configurable amount (default is 50%) of physical RAM.
		Depending on the amount you use, in most situations
		this means a process will not be killed while accessing
		pages but will receive errors on memory allocation as
		appropriate.

		Useful for applications that want to guarantee their
		memory allocations will be available in the future
		without having to initialize every page.

The overcommit policy is set via the sysctl `vm.overcommit_memory'.

The overcommit amount can be set via `vm.overcommit_ratio' (percentage)
or `vm.overcommit_kbytes' (absolute value).

The current overcommit limit and amount committed are viewable in
/proc/meminfo as CommitLimit and Committed_AS respectively.

Gotchas
-------

The C language stack growth does an implicit mremap. If you want absolute
guarantees and run close to the edge you MUST mmap your stack for the 
largest size you think you will need. For typical stack usage this does
not matter much but it's a corner case if you really really care

In mode 2 the MAP_NORESERVE flag is ignored. 


How It Works
------------

The overcommit is based on the following rules

For a file backed map
	SHARED or READ-only	-	0 cost (the file is the map not swap)
	PRIVATE WRITABLE	-	size of mapping per instance

For an anonymous or /dev/zero map
	SHARED			-	size of mapping
	PRIVATE READ-only	-	0 cost (but of little use)
	PRIVATE WRITABLE	-	size of mapping per instance

Additional accounting
	Pages made writable copies by mmap
	shmfs memory drawn from the same pool

Status
------

o	We account mmap memory mappings
o	We account mprotect changes in commit
o	We account mremap changes in size
o	We account brk
o	We account munmap
o	We report the commit status in /proc
o	Account and check on fork
o	Review stack handling/building on exec
o	SHMfs accounting
o	Implement actual limit enforcement

To Do
-----
o	Account ptrace pages (this is hard)

CommitLimit: Based on the overcommit ratio ('vm.overcommit_ratio'),
              this is the total amount of  memory currently available to
              be allocated on the system. This limit is only adhered to
              if strict overcommit accounting is enabled (mode 2 in
              'vm.overcommit_memory').
              The CommitLimit is calculated with the following formula:
              CommitLimit = ([total RAM pages] - [total huge TLB pages]) *
                             overcommit_ratio / 100 + [total swap pages]
              For example, on a system with 1G of physical RAM and 7G
              of swap with a `vm.overcommit_ratio` of 30 it would
              yield a CommitLimit of 7.3G.
              For more details, see the memory overcommit documentation
              in vm/overcommit-accounting.
Committed_AS: The amount of memory presently allocated on the system.
              The committed memory is a sum of all of the memory which
              has been allocated by processes, even if it has not been
              "used" by them as of yet. A process which malloc()'s 1G
              of memory, but only touches 300M of it will show up as
	      using 1G. This 1G is memory which has been "committed" to
              by the VM and can be used at any time by the allocating
              application. With strict overcommit enabled on the system
              (mode 2 in 'vm.overcommit_memory'),allocations which would
              exceed the CommitLimit (detailed above) will not be permitted.
              This is useful if one needs to guarantee that processes will
              not fail due to lack of memory once that memory has been
              successfully allocated.

#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <stdint.h>

int main(void)
{
    char *temp_malloc = NULL;
    uint64_t i=0;

    printf("wait malloc\n");
    sleep(10);
    printf("start malloc\n");
        
    temp_malloc=malloc(1024*1024*30);
    if(temp_malloc ==NULL)
    {   
        printf("malloc fail\n");
        return 0;
    }   
        
    for(i=0; i< 1024*1024*30; i++);
    {   
        temp_malloc[i] = i&0xff;
    }   
        
    printf("wr done\n");
    sleep(100);
        
    printf("free malloc\n");
    free(temp_malloc);
    return 0;
}

 如上程序以为能分配到物理内存，可是实际上free命令结果看不到used memory增加

meminfo结果对比，申请的30M都被over committed 了，所以如上程序也没有真的touch到内存？ 

 

/ # busybox pmap  2655
2655: {no such process} /mnt/mmc/sdcard/largemem
00010000       4K r-xp  /mnt/mmc/sdcard/largemem
00020000       4K r--p  /mnt/mmc/sdcard/largemem
00021000       4K rw-p  /mnt/mmc/sdcard/largemem
00022000     132K rw-p  [heap]
b50d0000   30724K rw-p    [ anon ]
b6ed1000     892K r-xp  /lib/libc-2.30.so
b6fb0000      60K ---p  /lib/libc-2.30.so
b6fbf000       8K r--p  /lib/libc-2.30.so
b6fc1000       4K rw-p  /lib/libc-2.30.so
b6fc2000      12K rw-p    [ anon ]
b6fc5000     100K r-xp  /lib/ld-2.30.so
b6feb000       8K rw-p    [ anon ]
b6fed000       4K r--p  /lib/ld-2.30.so
b6fee000       4K rw-p  /lib/ld-2.30.so
bed26000     132K rw-p  [stack]
bedd7000       4K r-xp  [sigpage]
bedd8000       4K r--p  [vvar]
bedd9000       4K r-xp  [vdso]
ffff0000       4K r-xp  [vectors]
mapped: 32108K

pmap的结果也显示出了30M的内存使用

google 查到有人用mlock 来实现防止被overcommit掉

#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <stdint.h>
#include <sys/mman.h>

int main(void)
{
    char *temp_malloc = NULL;
    uint64_t i=0;

    printf("wait malloc\n");
    sleep(10);
    printf("start malloc\n");
        
    temp_malloc=malloc(1024*1024*30);
    if(temp_malloc ==NULL)
    {   
        printf("malloc fail\n");
        return 0;
    }   
    if(mlock(temp_malloc, 1024*1024*30))
    {   
        printf("mlock fail\n");
        free(temp_malloc);
        return 0;
    }   
        
    printf("wr done\n");
    sleep(100);
        
    printf("free malloc\n");
    free(temp_malloc);
    return 0;
}

测试结果used 果然增加了30M

/ # free
             total         used         free       shared      buffers
Mem:        352328        42864       309464            0           40
-/+ buffers:              42824       309504
Swap:            0            0            0
/ # freestart malloc
wr done

             total         used         free       shared      buffers
Mem:        352328        73576       278752            0           40
-/+ buffers:              73536       278792
Swap:            0            0            0

至于为什么真正使用了malloc的memory也会被overcommit掉的原因，需要另开一篇认真调研下