How to implement growable memory-mapped files (GMMFs) in NTFS

Q In your article, "A File System for the 21st Century: Previewing the Windows NT 5.0 File System" (MSJ, November 1998), you mentioned the new sparse files feature. I was wondering how (or if) you could use this feature to create growable memory-mapped files.

Joseph Ryan
Somerset, NJ

A Yes, when I first started using sparse files, I immediately thought about how they could be used for growable memory-mapped files (GMMFs). Memory-mapped files (MMFs) are one of my favorite Windows features. For those of you who need a refresher, MMFs allow you to use memory addressing techniques to access the contents of your disk files. This makes it incredibly easy to manipulate the contents of a file, since accessing memory is always easier than doing buffered file I/O. A big problem with MMFs is that they are not growable. In other words, when you call CreateFileMapping to create an MMF, you must tell it exactly how much disk space to allocate.
Let's say that you want to create an MMF to store recorded audio data. As the user speaks, you want to write the digital audio data into a memory buffer and have that buffer backed by a file on the disk. There is a really easy and very efficient way to implement this in your code. The problem is that you don't know how long the user will speak before pressing the Stop button. You may need a file large enough for 5 minutes of data—or 5 hours. So, if you really want to use MMFs in your application, you must call CreateFileMapping, passing the maximum size you think you'll ever need and write the audio data as it comes in. When the user presses the Stop button, you would then close the file-mapping object and shrink the size of the file down by calling SetFilePointer followed by SetEndOfFile.
As you probably guessed, most applications (hopefully) will not use MMFs to implement this because it means an enormous amount of disk space is allocated while the audio data is being written. The likelihood of the data file staying its maximum size once the user stops recording is very slim. This storage problem is well-known to people who use MMFs. Way back when, I created my own GMMF mechanism that allowed a developer to work with MMFs, allocating disk storage only as needed. (See "Add Growable Memory-Mapped Files to Your App with Our Exclusive GMMF API," in the October 1995 issue of MSJ.) The code that I published in that article is still appropriate if you are working with Windows 9x or Windows NT 4.0 or earlier. In fact, my GMMF code is good for all file systems other than Windows NT File System (NTFS) 5.0. If you're designing an application for Windows 2000 and you know that your customer will be saving their data on an NTFS 5.0 partition, then you can use the new sparse file feature to make GMMFs simple to implement.
 

/*****************************************************************************
Module name: GMMF.cpp
Written by: Jeffrey Richter
Purpose:    Function prototypes for using growable memory-mapped files.
*****************************************************************************/

#define _WIN32_WINNT 0x0500 // Sparse files require Windows 2000
#include <windows.h>
#include <WinIoCtl.h>


//


// This class makes it easy to work with memory-mapped sparse files
class CGMMF {
   HANDLE m_hfilemap;      // File-mapping object
   PVOID  m_pvFile;        // Address to start of mapped file

public:
   // Creates a GMMF and maps it in the process's address space.
   CGMMF(HANDLE hFile, UINT_PTR cbFileSizeMax);

   // Closes a GMMF
   ~CGMMF() { ForceClose(); }

   // GMMF to BYTE cast operator returns address of first byte 
   // in the memory-mapped sparse file. 
   operator PBYTE() const { return((PBYTE) m_pvFile); }

   // Allows you to explicitly close the GMMF without having
   // to wait for the destructor to be called.
   VOID ForceClose();

public:
    // Static method that resets a portion of a file back to
    // zeroes (frees disk clusters)
    static BOOL SetToZero(HANDLE hfile, 
        UINT_PTR cbOffsetStart, UINT_PTR cbOffsetEnd);
};


//


CGMMF::CGMMF(HANDLE hfile, UINT_PTR cbFileSizeMax) {
    // Initialize to NULL in case something goes wrong
    m_hfilemap = m_pvFile = NULL;

    // Make the file sparse
    DWORD dw;
    BOOL fOk = ::DeviceIoControl(hfile, FSCTL_SET_SPARSE, 
        NULL, 0, NULL, 0, &dw, NULL);

    if (fOk) {
        // Create a file-mapping object
        m_hfilemap = ::CreateFileMapping(hfile, NULL, 
            PAGE_READWRITE, 0, cbFileSizeMax, NULL);

        if (m_hfilemap != NULL) {
            // Map the file into the process's address space
            m_pvFile = ::MapViewOfFile(m_hfilemap, 
                FILE_MAP_WRITE | FILE_MAP_READ, 0, 0, 0);
        } else {
            // Failed to map the file, cleanup
            ForceClose();
        }
    }
}


//


VOID CGMMF::ForceClose() {
    // Cleanup everything that was done sucessfully
    if (m_pvFile   != NULL) { 
        ::UnmapViewOfFile(m_pvFile); 
        m_pvFile = NULL; 
    }
    if (m_hfilemap != NULL) { 
        ::CloseHandle(m_hfilemap);   
        m_hfilemap = NULL; 
    }
}


//


BOOL CGMMF::SetToZero(HANDLE hfile, 
    UINT_PTR cbOffsetStart, UINT_PTR cbOffsetEnd) {

    // NOTE: This function does not work if this file is memory-mapped.
    DWORD dw;
    FILE_ZERO_DATA_INFORMATION fzdi;
    fzdi.FileOffset.QuadPart = cbOffsetStart;
    fzdi.BeyondFinalZero.QuadPart = cbOffsetEnd + 1;
    return(::DeviceIoControl(hfile, FSCTL_SET_ZERO_DATA, 
        (LPVOID) &fzdi, sizeof(fzdi), NULL, 0, &dw, NULL));
}


//


int WINAPI WinMain (HINSTANCE, HINSTANCE, LPSTR, int ) {

    char szPathname[] = "C:\\GMMF.";

    // Create the file
    HANDLE hfile = CreateFile(szPathname, GENERIC_READ | GENERIC_WRITE, 
        0, NULL, CREATE_ALWAYS, FILE_ATTRIBUTE_NORMAL, NULL);

    // Create a GMMF using the file (set the maximum size here too)
    CGMMF gmmf(hfile, 10 * 1024 * 1024);

    // Read bytes from the file (0s are returned)
    for (int x = 0; x < 10 * 1024 * 1024; x += 1024) {
        if (gmmf[x] != 0) DebugBreak();
    }

    // Write bytes to the file (clusters are allocated as necessary).
    for (x = 0; x < 100; x++) {
        gmmf[8 * 1024 * 1024 + x] = x;
    }

    // These lines just prove to us what's going on
    DWORD dw = GetFileSize(hfile, NULL);
    // This returns the logical size of the file.

    // Get the actual number of bytes allocated in the file
    dw = GetCompressedFileSize(szPathname, NULL);
    // This returns 0 because the data has not been written to the file yet.
    
    // Force the data to be written to the file
    FlushViewOfFile(gmmf, 10 * 1024 * 1024);

    // Get the actual number of bytes allocated in the file
    dw = GetCompressedFileSize(szPathname, NULL);
    // This returns the size of a cluster now

    // Normally the destructor causes the file-mapping to close.
    // But, in this case, we wish to force it so that we can reset 
    // a portion of the file back to all zeroes.
    gmmf.ForceClose();

    // We call ForceClose above because attempting to zero a portion of the 
    // file while it is mapped, causes DeviceIoControl to fail with error 
    // code 0x4C8 (ERROR_USER_MAPPED_FILE: "The requested operation cannot 
    // be performed on a file with a user-mapped section open.")
    CGMMF::SetToZero(hfile, 0, 2 * 1024 * 1024);

    // We no longer need access to the file, close it.
    CloseHandle(hfile);

    return(0);
}

 End of File /

   The code shows how to create a GMMF using sparse files. To make things easier, I created a CGMMF class that encapsulates the details. The class member functions do all the sparse file stuff that I explained in the November 1998 NTFS article, so I won't go into the fine points here. Instead, I'll just concentrate on the WinMain function.
When the process starts running, I first call CreateFile to create a new file on my NTFS 5.0 disk partition. This is a normal, ordinary file. Then I construct a CGMMF object, passing its constructor the file handle and the maximum size to which I'll allow this file to grow. The constructor tells NTFS 5.0 to make the file a sparse file and creates a file-mapping object that is the maximum size. Note that my code allows the file to grow to 10MB, but since the file is sparse, the file on the disk will actually contain zero bytes! If this were not a sparse file, the file on the disk would actually be 10MB.
Once the CGMMF object is constructed, I try to read some bytes from within the file. I do this simply by creating a PBYTE cast operator method in my CGMMF class. When you read bytes from a sparse file, the file system automatically knows to return 0s for bytes that don't actually exist in the file. So, my For loop doesn't generate any access violations, and when the loop completes, the file on the disk still contains zero bytes.
After the reading is finished, I have another loop that writes 100 bytes, starting 8MB into the disk file. Again, no access violations occur when I execute this loop. When the loop completes, the file will no longer be zero bytes in length; instead, it will now have just enough disk storage allocated to it to hold the 100 bytes. Note that the first 8MB of this file still requires no disk storage at all. This is extremely efficient use of disk space.
The next few lines of code show how to get the logical and physical size of the file. Simply examine the results in the debugger to get a better understanding of what's going on.
Toward the end of WinMain, you see another concept demonstrated. Sometimes the user may want to delete part of a file. For example, the user may choose to delete the first 10 minutes of the audio data file. To delete this data, you can call my CGMMF class's SetToZero method. This method is static because it actually doesn't work on a GMMF at all. To delete a portion of the file, you must work with file handles, not file-mapping handles. In fact, to delete a portion of a sparse file, you must not have any file-mapping objects backed by that file or attempting to delete a portion of the file will fail.
So to delete a portion of the file, you must first call CGMMF's ForceClose method explicitly. This closes the file-mapping object so that the static SetToZero method can be called to delete a portion of the file, freeing up some disk space. At this point, if you want to continue working with the file as a GMMF, you can create another CGMMF object, passing the file handle just like I did at the beginning of WinMain. If you do not have a need to delete a portion of the file, then the CGMMF's destructor calls the ForceClose method when the object goes out of scope. 
One last thing. Remember the audio data example mentioned earlier? When the user presses the Stop button, you may want to cut the file's size back down to what you actually need. Trimming the end of a sparse file that contains zero bytes doesn't actually affect the disk space, but it is still a nice thing to do so that the Explorer and the command shell's DIR command report a more accurate file size to the user. To set the end of file marker for a file, you must call SetFilePointer and SetEndOfFile. Just like the call to SetToZero, these functions succeed only if there are no file-mapping objects for the file you're manipulating. This is another place where ForceClose comes in handy.

Reference:

http://www.microsoft.com/msj/0499/win32/win320499.aspx