Dynamic memory allocation example

How to use dynamic memory allocation

In the Simple memory locking example is explained that all memory must be allocated and claimed, for the entire lifetime of the RT-application, at startup time, before the RT-application is going to fulfill its RT requirements. If memory is allocated later on, this normally will result in pagefaults, and thus ruin the RT behavior of the application. 

Q: So, we cannot run C++ applications with dynamic memory allocation? 
A: Wrong! Dynamic memory allocation is possible, if:

• allocated memory, once committed and locked in RAM, is never given back to the kernel.

Q: How can this be achieved? 
A: All memory allocation routines are implemented inside Glibc. Glibc translates each memory allocation request to a call to:

• mmap(): mmap maps in a certain amount of memory into the virtual memory space of the process. mmap() is usually faster than sbrk() for smaller memory allocations, or
• sbrk(): sbrk increases (or decreases) the memory block assigned to the process by a given size.

Glibc offers interfaces that can be used to configure its behavior related to these calls. 
Glibc can be configured how much memory must be released before calling sbrk() to give memory back to the kernel. It can also be configured when sbrk() is used instead of mmap()
What we need to do is to get rid of the mmap calls, and to configure glibc to never give memory back to kernel, until the process terminates. (of course). 
We use this (badly documented) call for it: int mallopt (int param, int value) (it is defined in malloc.h.) When calling mallopt, the param argument specifies the parameter to be set, and value the new value to be set. Possible choices for param, as defined in malloc.h, are:

• M_TRIM_THRESHOLD: This is the minimum size (in bytes) of the top-most, releasable chunk that will cause sbrk() to be called with a negative argument in order to return memory to the system.
• M_TOP_PAD: This parameter determines the amount of extra memory to obtain from the system when a call to sbrk() is required. It also specifies the number of bytes to retain when shrinking the heap by calling sbrk() with a negative argument. This provides the necessary hysteresis in heap size such that excessive amounts of system calls can be avoided.
• M_MMAP_THRESHOLD: All chunks larger than this value are allocated outside the normal heap, using the mmap system call. This way it is guaranteed that the memory for these chunks can be returned to the system on free.
• M_MMAP_MAX: The maximum number of chunks to allocate with mmap. Setting this to zero disables all use of mmap.

More background information on how to use this mallopt() call can be found at this paper: 
http://www.usenix.org/publications/library/proceedings/als01/full_papers/ezolt/ezolt.ps 

The following example shows how we can create a pool of memory during startup, and lock it into memory. At startup a block of memory is allocated through the malloc() call. Prior to it Glibc will be configured such that it uses the sbrk() call to fulfill this allocation. After locking it, we can free this block of memory, knowing that it is not released to the kernel and still assigned to our RT-process.
We have now created a pool of memory that will be used by Glibc for dynamic memory allocation. We can new() and delete() as much as we want without being interfered by any page fault! Even if the system is fully stressed, and swapping is continuously active, the RT-application will never run into any page fault...

Another possibility is to use a separate malloc tool like the O(1) Memory Allocator together with a preallocated and locked buffer which is used as memory pool for the custom Memory Allocator. In that case all the new, delete, malloc and free operators have to be redirected to this custom Memory Allocator.

   #include <stdlib.h>
   #include <stdio.h>
   #include <sys/mman.h> // Needed for mlockall()
   #include <unistd.h> // needed for sysconf(int name);
   #include <malloc.h>
   #include <sys/time.h> // needed for getrusage
   #include <sys/resource.h> // needed for getrusage
   

   #define SOMESIZE (100*1024*1024) // 100MB
   

   int main(int argc, char* argv[])
   {
       // Allocate some memory
       int i, page_size;
       char* buffer;
       struct rusage usage;
       

       // Now lock all current and future pages from preventing of being paged
       if (mlockall(MCL_CURRENT | MCL_FUTURE ))
       {
           perror("mlockall failed:");
       }
       

       // Turn off malloc trimming.
       mallopt (M_TRIM_THRESHOLD, -1);
       

       // Turn off mmap usage.
       mallopt (M_MMAP_MAX, 0);
       

       page_size = sysconf(_SC_PAGESIZE);
       buffer = malloc(SOMESIZE);
       

       // Touch each page in this piece of memory to get it mapped into RAM
       for (i=0; i < SOMESIZE; i+=page_size)
       {
           // Each write to this buffer will generate a pagefault.
           // Once the pagefault is handled a page will be locked in memory and never
           // given back to the system.
           buffer[i] = 0;
           // print the number of major and minor pagefaults this application has triggered
           getrusage(RUSAGE_SELF, &usage);
           printf("Major-pagefaults:%d, Minor Pagefaults:%d\n", usage.ru_majflt, usage.ru_minflt);
       }
       free(buffer);
       // buffer is now released. As glibc is configured such that it never gives back memory to
       // the kernel, the memory allocated above is locked for this process. All malloc() and new()
       // calls come from the memory pool reserved and locked above. Issuing free() and delete()
       // does NOT make this locking undone. So, with this locking mechanism we can build C++ applications
       // that will never run into a major/minor pagefault, even with swapping enabled.
       

       //<do your RT-thing>
       

       return 0;
   }