理解Python中mmap : memory-mapped file

• Python library : mmap

  Memory-mapped file objects behave like both bytearray and like file objects. You can use mmap objects in most places where bytearray are expected:

  • you can use the re module to search through a memory-mapped file
  • you can change a single byte by doing obj[index] = 97
  • you can change a subsequence by assigning to a slice : obj[i_1:i_2] = b"..."
  • you can read and write data starting at the current file position and seek() through the file to different positions

  A memory-mapped file is created by the mmap constructor, chich is different on Unix and on Windows. In either case you must provide a file descriptor for a file opened for update.

  If you wish to map an existing Python file object, use its fileno() method to obtain the correct value for the fileno parameter.

  Otherwise, you can open the file using the os.open() function, which returns a file descriptor directly.

  If you want to create a memory-mapping for a writable, buffered file, you should flush() the file first . This is necessary to ensure that local modifications to the buffers are actually available to the mapping.

  For both the Unix and Windows versions of the constructor, access may be specified as an optional keyword parameter. Access accepts one of four values:

  1. ACCESS_READ
  2. ACCESS_WRITE
  3. ACCESS_COPY(read-only)
  4. ACCESS_DEFAULT (to defer to prot.access)

  To map anonymouse memory, -1 should be passes as the fileno along with the length.

  mmap can also be used as a context manager in a with statement.

• Memory-mapped file objects support methods

  • close()

  • closed

  • find(sub[,start[,end]])

  • flush([offset[,size]])

    Flushes changes made to the in-memory copy of a file back to disk.

    Without use of this call there is no guarantee that changes are written back before the object is destroyed.

    if offset and size are specified, only changes to the given range of bytes will be flushed to disk; otherwise, the whole extent of the mapping is flushed.

    offset must be a multiple of the PAGESIZE or ALLOCATIONGRANULARITY

  • madvise(option[, start[, length]])

  • move(dest, src, count)

  • read([n])

  • read_byte()

  • readline()

  • resize(newsize)

  • rfind(sub[, start[, end]])

  • seek(pos[, whence])

  • size()

  • tell()

  • write(bytes)

  • write_byte(byte)

• MADV_*Constants

  mmap.MADV_NORMAL

  mmap.MADV_RANDOM

  mmap.MADV_SEQUENTIAL

  mmap.MADV_WILLNEED

  mmap.MADV_DONTNEED

  mmap.MADV_REMOVE

  mmap.MADV_DONTFORK

  mmap.MADV_DOFORK

  mmap.MADV_HWPOISON

  mmap.MADV_MERGEABLE

  mmap.MADV_UNMERGEABLE

  mmap.MADV_SOFT_OFFLINE

  mmap.MADV_HUGEPAGE

  mmap.MADV_NOHUGEPAGE

  mmap.MADV_DONTDUMP

  mmap.MADV_DODUMP

  mmap.MADV_FREE

  mmap.MADV_NOSYNC

  mmap.MADV_AUTOSYNC

  mmap.MADV_NOCORE

  mmap.MADV_CORE

  mmap.MADV_PROTECT

  These options can be passed to mmap.madvise().

• PyMOTW-3 : mmap – Memory-map files

  Memory-map files instead of reading the contents directly.

  Memory-mapping a file uses the OS virtual memory system to access the data on the file system directly, instead of using normal I/O fucntions.

  Memory-mapping typically improves I/O performance because it does not involve a separate system call for each access and it does not require copying data between buffers - the memory is accessed directly by both the kernel and the user application.

  Memory-mapped files can be treated as mutable strings or file-like objects, depending on the need. A mapped file supports the expected file API methods, such as close(), flush(), read(), readline(), seek(), tell(), and write(). It also supports the string API, with features such as slicing and methods like find().

  All of the examples use the text file 1 orem.txt

• Pythontic.com

  A memory-mapped file is a region of memory that is mapped to a disk file.

  The mechanism involved in memory-mapped files is the same mechanism used by OS for implementing virtual memory.

  With memory mapped files programs access the data directly from the memory rather than using any I/O routines.

  When files are dealt as memory mapped files, the contents of the files can be accessed in the same way a region of memory is accessed, from a Python Program.

  To create a memory mapped file the following information needs to be provided to the mmap constructor:

  1. A file descriptor, which got by file.fileno() which file is got by os.open()
  2. Number of bytes of the file to be mapped
  3. Offset of the bytes from the beginning of the file
  4. Access level - ACCESS_READ for read-only, ACCESS_WRITE for read and write, ACCESS_COPY for copy-on-write

  When the file descriptor is specified as -1 an anonymous mmap is created that is a memory region, which is not backed by a disk file.