c++处理大量数据

用内存映射应该行的。内存映射就是把一个文件当作一块内存区域用。
类别: 内存映射文件
题目:对一个文件使用内存映射文件
Demo:
1:创建或打开一个文件内核对象:
// Open the file for reading and writing.
HANDLE hFile = CreateFile(pszPathname, GENERIC_WRITE | GENERIC_READ, 0,
NULL, OPEN_EXISTING, FILE_ATTRIBUTE_NORMAL, NULL);
// 由于hFile即使为INVALID_HANDLE_VALUE,下面的CreateFileMapping仍然可以正常运行,
// 所以这里一定要对hFile进行检查!
if (hFile == INVALID_HANDLE_VALUE) {
chMB("File could not be opened.");
return(FALSE);
}
2:创建一个文件映射内核对象:
// Get the size of the file (I assume the whole file can be mapped).
DWORD dwFileSize = GetFileSize(hFile, NULL);
// Create the file-mapping object. The file-mapping object is 1 character
// bigger than the file size so that a zero character can be placed at the
// end of the file to terminate the string (file). Because I don't yet know
// if the file contains ANSI or Unicode characters, I assume worst case
// and add the size of a WCHAR instead of CHAR.
HANDLE hFileMap = CreateFileMapping(hFile, NULL, PAGE_READWRITE,
0,
dwFileSize + sizeof(WCHAR), // 如果该文件小于设定的大小,本函数将扩展该文件的大小,
// 使磁盘上的文件变大。这样当以后将该文件作为内存映射
// 文件使用时,物理存储器就已经存在了。
NULL f// 这个文件映射对象的名字用于与其他进程共享该对象,这里我们还用不到。
);
if (hFileMap == NULL) {
chMB("File map could not be opened.");
CloseHandle(hFile);
return(FALSE);
}
3:将文件数据映射到进程的地址空间:
当创建了一个文件映射对象之后,仍然必须让系统为文件的数据保留一个地址空间区域,
并将文件的数据作为映射到该区域的物理存储器进行提交。
// Get the address where the first byte of the file is mapped into memory.
// the return value is the starting address of the mapped view:
PVOID pvFile = MapViewOfFile(hFileMap, FILE_MAP_WRITE, 0, 0, 0);
if (pvFile == NULL) {
chMB("Could not map view of file.");
CloseHandle(hFileMap);
CloseHandle(hFile);
return(FALSE);
}
4:既然我们通过pvFile得到了映象视图的起始地址,那么可以对视图做一些操作了:
ANSI版本:
PSTR pchANSI = (PSTR) pvFile;
UNICODE版本:
PWSTR pchUnicode = (PWSTR) pvFile;
5:从进程的地址空间中撤销文件数据的映象:
// Clean up everything before exiting.
UnmapViewOfFile(pvFile);
6:关闭文件映射对象和文件对象:
CloseHandle(hFileMap);
CloseHandle(hFile);
Definition:
HANDLE CreateFileMapping(
HANDLE hFile, // handle to file
LPSECURITY_ATTRIBUTES lpAttributes, // security
DWORD flProtect, // protection
DWORD dwMaximumSizeHigh, // high-order DWORD of size
DWORD dwMaximumSizeLow, // low-order DWORD of size
LPCTSTR lpName // object name
);
LPVOID MapViewOfFile(
HANDLE hFileMappingObject, // handle to file-mapping object
DWORD dwDesiredAccess, // access mode
DWORD dwFileOffsetHigh, // high-order DWORD of offset
DWORD dwFileOffsetLow, // low-order DWORD of offset
SIZE_T dwNumberOfBytesToMap // number of bytes to map
);
BOOL UnmapViewOfFile(
LPCVOID lpBaseAddress // starting address
);
Tips:
也可以尽量早地把对象关闭,以消除资源泄漏的可能性,如:
HANDLE hFile = CreateFile(...);
HANDLE hFileMapping = CreateFileMapping(hFile,...);
CloseHandle(hFile);
PVOID pvFile = MapViewOfFile(hFileMapping,...);
CloseHandle(hFileMapping);
// use the memory-mapped file.
UnmapViewOfFile(pvFile);