Linux部分转自http://blog.csdn.net/songguozhi/article/details/3085841,未验证
Windows部分转自MSDN
Linux
Low-level I/O 和 File System Interface
1. 为什么需要使用 low-level I/O,glibc 里面提到了一些情形,对大量二进制数据进行操作,某些文件上的操作只能通过 descriptor(设置一些权限,如锁),
把一些资源传递给子进程。使用 low-level I/O 的很多函数是把前面 stream 的
那个 f 去掉了,如
int open ( const char *filename, int flags[, mode_t mode])一些不同在于,这些函数对 file descriptor 操作(就是一个 int)。文件打开的权限
int creat ( const char *filename, mode_t mode)
int close ( int filedes)
ssize_t read ( int filedes, void *buffer, size_t size)
ssize_t pread ( int filedes, void *buffer, size_t size, off_t offset)
ssize_t write ( int filedes, const void *buffer, size_t size)
ssize_t pwrite ( int filedes, const void *buffer, size_t size, off_t offset)
off_t lseek ( int filedes, off_t offset, int whence)
可以为 O_RDONLY,O_WRONLY,O_RDWR,但是 POSIX.1 下面最好用 O_READ,
O_WRITE,O_EXEC,这是所谓的 access modes,另外还有 Open-time flags,如
O_CREAT(不存在就创建),O_EXCL(和前者同时使用表示存在即报错),
O_NONBLOCK(打开某些设备如串口时避免等待很长时间,之后如果 I/O 需要 block 可
利用 fcntl 更改),O_NOCTTY(针对终端设备,不懂 @@),O_IGNORE_CTTY(同),
O_NOLINK(打开的文件是符号连接本身),O_NOTRANS(不进行 translate 工作),
O_TRUNC(截断原来的东西,ms w 就是干这个事情了),O_SHLOCK(共享锁,允许
其他的进程读),O_EXLOCK(排除性锁,不允许其他的读写),另外有一些 I/O
operating modes,如 O_APPEND(指向文件尾),O_ASYNC(允许异步读写,通
过 SIGIO 信号),O_FSYNC(同步写入文件),O_NOATIME(不更新 access time)。
其中 p 类型的读写表示根据到文件头的偏移进行的,另外 create 函数是 obsolete 的。
这里没有类似的 tell 函数,但是可以用 lseek (desc, 0, SEEK_CUR) 获得。
可以使用
FILE * fdopen ( int filedes, const char *opentype)建立一个 stream 关联到该 descriptor 上,而
int fileno (FILE *stream)将返回某个 stream 的 descriptor,通常 stdin、stdout 和 stderr 的 descriptor 可以用
int fileno_unlocked (FILE *stream)
STDIN_FILENO、STDOUT_FILENO、STDERR_FILENO 表达。可见同一个 descriptor
上可以建立不同的 stream,这时两个 streams 其实是 linked 在一起,即具有相同的当前
文件指针(不是说 FILE*,这其实是 stream 的地址)等等。另外可以用
int dup ( int old)产生 linked 在一起的 descriptor(编号不同),后者使用 new 原来的编号但使用的 old 的
int dup2 ( int old, int new)
信息;前者返回的最小的未使用的 descriptor 作为 old 的“复制品”。后面讲述 fcntl() 会进一
步解释这两个函数的。清理一个 stream 可以使用
int fclean (FILE *stream)其作用基本和 fflush() 近似。
2. 快速 scatter-gather I/O,解决的问题是把从一个地方读出的数据快速传输到几个不同的
buffer 中,数据是一个一个 buffer 填满,然后下一个,直到输入结束或填满了所有的 buffer,
描述一个 buffer 使用一个 struct iovec,其成员 void *iov_base 描述缓存区地址,size_t iov_len
是缓存区大小,使用(sys/uio.h)
ssize_t readv ( int filedes, const struct iovec *vector, int count)从 descriptor 读入到 iovec 指向的结构体所描述的缓存中,该区域一共包含 count 个缓存区。
写入使用
ssize_t writev ( int filedes, const struct iovec *vector, int count)
3. 现代操作系统多支持所谓 mmap(memory mapped)的方式快速读写文件,往往一个
page 存储一段数据,页面大小可以用
size_t page_size = (size_t) sysconf (_SC_PAGESIZE);获得,然后将文件内容映射到一个内存区域(sys/mman.h)
void * mmap ( void *address, size_t length, int protect, int flags, int filedes, off_t offset)读写内存后就可以把改动存储到文件中,读写完后需要
void * mremap ( void *address, size_t length, size_t new_length, int flag)
int munmap ( void *addr, size_t length)如果需要显示将读写同步,应调用
int msync ( void *address, size_t length, int flags)下面我们解释一下相关的参数,mmap 将 fildes 对应 offset 开始 长度为 length 的数据
映射到 address 处(希望在这里 @@),可以为 NULL,然后会分配的,protect 是
权限,有 PROT_READ、PROT_WRITE、PROT_EXEC。而 flags 表明使用的一些
其他的 MAP_PRIVATE(不写回文件),MAP_SHARED(写回文件,这样其他打开
该文件的程序会立即看见),MAP_FIXED(使用 address 否则 fail),MAP_ANON
(仅仅创建一个内存块,不与文件关联)。使用 munmap 时只会释放正好一个或多个
pages,不会释放一部分(尽管可以这么调用)。在 msync 里面的 flags 可以为 MS_SYNC
(保证写到磁盘上),MS_ASYNC(可以开始同步,但理解返回,不会 block)。
另外为了更高效的利用这种内存,可以提出一些建议
int madvise ( void *addr, size_t length, int advice)其中,advice 可以是 MADV_NORMAL(正常处理),MADV_RANDOM(会被随机访问),
MADV_SEQUENTIAL(会被顺序访问),MADV_WILLNEED(不懂),MADV_DONTNEED(ft...)。
这部分看点 paging 再看比较合适 -.-
4. 需要检查多个输入输出是否有数据到达可以使用
int select (int nfds, fd_set *read-fds, fd_set *write-fds, fd_set *except-fds, struct timeval *timeout)
其中使用 nfds 表示有多少个 descriptor 被监视,他们放在 read-fds、write-fds 和 except-fds 中,
并用 timeout 所指向表示一个超时(过后不待 @@),宏 FD_SETSIZE 决定一个 descriptor 的集合
的大小,我们需要 fd_set 存放 descriptors,可以用以下宏操纵:
void FD_ZERO (fd_set *set)
void FD_SET (int filedes, fd_set *set)
void FD_CLR (int filedes, fd_set *set)
int FD_ISSET (int filedes, const fd_set *set)
依次表示一个空的 fd_set、添加一个 descriptor、清除一个 descriptor、测试是否属于该 fd_set。
如果超时,select 返回 0,如果出现错误、接收到 signal 则返回。
5. 同步 I/O 常用下面的函数
int sync ( void)
int fsync ( int fildes)
int fdatasync ( int fildes)
6. 异步输入输出,不同在于函数会立即返回,操作排在队列中回自动进行,其控制过程用
struct aiocb 决定,其包含控制某个读写的信息,如 descriptor、偏移等等。下面的操作
进行读写等等
int aio_read ( struct aiocb *aiocbp)这些函数分别实现了读、写、批量读写、侦测错误、查看返回值、同步数据、暂停、取消
int aio_write ( struct aiocb *aiocbp)
int lio_listio ( int mode, struct aiocb * const list[], int nent, struct sigevent *sig)
int aio_error ( const struct aiocb *aiocbp)
ssize_t aio_return ( const struct aiocb *aiocbp)
int aio_fsync ( int op, struct aiocb *aiocbp)
int aio_suspend ( const struct aiocb * const list[], int nent, const struct timespec *timeout)
int aio_cancel ( int fildes, struct aiocb *aiocbp)
这些功能。同时还可以用 struct aioinit 结构进行优化,这需要首先用
void aio_init ( const struct aioinit *init)然后用户设定的策略就可以被应用到后面的 AIO 上了。
6. fcntl 的功能
int fcntl ( int filedes, int command, ...)有很多,如 F_DUPFD 复制 discriptor、F_GETFD(获得 descriptor 的 flags)、F_GETFL
(获得打开文件的 flags)、F_GETLK(获得文件的 lock)、F_GETOWN(获取进程 gid),
对应的 GET 都有 SET 版本,多了一个 F_SETLKW 表示等待完成才返回。比如前面的 dup
可以 fcntl (old, F_DUPFD, 0) 实现。文件 descriptor flags 中有 FD_CLOEXEC 表示在
exec 之后是否关闭。
7. 一些其他的控制使用
int ioctl ( int filedes, int command, ...)完成,如控制终端的行为等。
8. 关于目录的几个函数
char * getcwd ( char *buffer, size_t size)这三个都是获得目录,中间一个最好不用,其实使 buffer 为 NULL 等价第三个,都是
char * getwd ( char *buffer)
char * get_current_dir_name ( void)
malloc 出来需要 free 的。下面的是用于改变当前目录:
int chdir ( const char *filename)注意这里面很多 f 开头的却不是 stream 都是用的 descriptor。
int fchdir ( int filedes)
为了处理一般的目录里面的东西,需要一个结构 dirent 描述,里面包含了诸如 d_name
名称,d_fileno(inode),d_type(类型)这些东西。其中 d_type 是 mode_t 类型,
可以用
int IFTODT (mode_t mode)互相转化(宏)。首先必须打开一个目录,这使用的是
mode_t DTTOIF ( int dtype)
DIR * opendir ( const char *dirname)它创建了一个 stream,而
int dirfd (DIR *dirstream)将返回它的 descriptor,然后用
struct dirent * readdir (DIR *dirstream)读入需要的东西(directory entry),后者是 thread-safe 的。读完后用
int readdir_r (DIR *dirstream, struct dirent *entry, struct dirent **result)
int closedir (DIR *dirstream)关闭这个流。如果需要以某种特定的方式,如依照字母序来访问一个 directory 里面的
东西,就可以使用
int scandir ( const char *dir, struct dirent ***namelist, int (*selector) ( const struct dirent *), int (*cmp) ( const void *, const void *))其中可以用来作为 cmp 的为
int alphasort ( const void *a, const void *b)这类函数,一般使用 scandir 扫描一个目录,用 select 筛选,cmp 排序,由于 dirent
int versionsort ( const void *a, const void *b)
是用 malloc 分配的事后需要释放。
另外有
int ftw ( const char *filename, __ftw_func_t func, int descriptors)其中
int nftw ( const char *filename, __nftw_func_t func, int descriptors, int flag)
typedef int (*__ftw_func_t) ( const char *, const struct stat *, int)其用途在于 ftw()(file tree walk)将从 filename 开始遍历整棵树,将 func 作用在每一个节点上,
typedef int (*__nftw_func_t) ( const char *, const struct stat *, int, struct FTW *)
其中 stat 是其 file stat,另一个 int 是文件类型。descriptors 限制最多使用的 descriptors 数目。
而 nftw 稍微不同一点在于后面的 flag 可以 OR 到所有的 func 调用的 int 上去。
9. 关于链接,可以用下面的函数
int link ( const char *oldname, const char *newname)前者用于建立硬链接,后者是符号链接,可以用
int symlink ( const char *oldname, const char *newname)
int readlink ( const char *filename, char *buffer, size_t size)读出符号链接的目标文件。函数
char * canonicalize_file_name ( const char *name)返回不含有 . 或者 .. 的典则文件名。而
char * realpath ( const char *restrict name, char *restrict resolved)类似,只是放到 resolved 里面(除非为 NULL 就和前者一样了)。
删除文件就是一个 link 的反过程
int unlink ( const char *filename)而这是删除目录,
int rmdir ( const char *filename)ISO C 里面定义了
int remove ( const char *filename)它对文件目录都 work。
int rename ( const char *oldname, const char *newname)重命名文件,
int mkdir ( const char *filename, mode_t mode)创建目录, mode_t 声明了权限,
10. 文件权限常用 struct stat 表示,里面含有多数我们关心的信息,如 st_mode(类型、权限),
st_ino(inode),st_nlink(硬连接数),属主,大小,访问、修改、修改属性时间等。一般使用
int stat ( const char *filename, struct stat *buf)这其中某个函数获得,最后一个是不会 follow 符号链接的。有一系列的 macro 能够检测我们需要的
int fstat ( int filedes, struct stat *buf)
int lstat ( const char *filename, struct stat *buf)
类型等,如
int S_ISDIR (mode_t m)我们可以用以下函数更改文件属主,
int S_ISCHR (mode_t m)
int S_ISBLK (mode_t m)
int S_ISREG (mode_t m)
int S_ISFIFO (mode_t m)
int S_ISLNK (mode_t m)
int S_ISSOCK (mode_t m)
int S_TYPEISMQ ( struct stat *s)
int S_TYPEISSEM ( struct stat *s)
int S_TYPEISSHM ( struct stat *s)
int chown ( const char *filename, uid_t owner, gid_t group)可以用下面函数设置 umask
mode_t umask (mode_t mask)调用该函数设置了 umask 同时会返回原来的 umask。可以直接用
mode_t getumask ( void)获得 umask,
int chmod ( const char *filename, mode_t mode)改变文件权限,可以用
int fchmod ( int filedes, int mode)
int access ( const char *filename, int how)测试权限。如果只需要文件的时间信息,可以用
int utime ( const char *filename, const struct utimbuf *times)该结构 times 中含有访问时间和修改时间,
int utimes ( const char *filename, struct timeval tvp[2])将结果存在 sys/time.h 中的 timeval 类型中,l 表示不 follow 符号链接。
int lutimes ( const char *filename, struct timeval tvp[2])
int futimes ( int *fd, struct timeval tvp[2])
使用
int truncate ( const char *filename, off_t length)将会截断文件,使用
int ftruncate ( int fd, off_t length)
int mknod ( const char *filename, int mode, int dev)创建一些设备相关的文件,这需要更深入的了解。
11. 下面是一些创建临时文件的函数,可以避免使用固定文件可能造成的问题
FILE * tmpfile ( void)其中 tmpfile 直接返回了一个 stream,后面的 nam(e) 系列仅仅返回文件名,
char * tmpnam ( char *result)
char * tmpnam_r ( char *result)
char * tempnam ( const char *dir, const char *prefix)
char * mktemp ( char *template)
int mkstemp ( char *template)
长度由 L_tmpnam 控制,tempnam 最强大(设定路径以及前缀),而 mktemp
依照 template 中 XXXX(以之结尾)的个数产生对应的字符串,而 s 版本返回的
是 descriptor 了。
Windows部分
The I/O functions read and write data to and from files and devices. File I/O operations take place in text mode or binary mode. The Microsoft run-time library has three types of I/O functions:
-
Stream I/O functions treat data as a stream of individual characters.
-
Low-level I/O functions invoke the operating system directly for lower-level operation than that provided by stream I/O.
-
Console and port I/O functions read or write directly to a console (keyboard and screen) or an I/O port (such as a printer port).
Note Because stream functions are buffered and low-level functions are not, these two types of functions are generally incompatible. For processing a particular file, use either stream or low-level functions exclusively.
微软的运行时库,提供了三种类型的I/O操作函数
Stream I/O
These functions process data in different sizes and formats, from single characters to large data structures. They also provide buffering, which can improve performance. The default size of a stream buffer is 4K. These routines affect only buffers created by the run-time library routines, and have no effect on buffers created by the operating system.
Routine | Use | .NET Framework equivalent |
---|---|---|
Clear error indicator for stream | Not applicable. To call the standard C function, use PInvoke. For more information, see Platform Invoke Examples. | |
Close stream | System::IO::Stream::Close, System::IO::BinaryReader::Close, System::IO::BinaryWriter::Close, System::IO::TextReader::Close, System::IO::TextWriter::Close, System::IO::StringReader::Close, System::IO::StringWriter::Close, System::IO::StreamReader::Close, System::IO::StreamWriter::Close | |
Close all open streams except stdin, stdout, and stderr | System::IO::Stream::Close, System::IO::BinaryReader::Close, System::IO::BinaryWriter::Close, System::IO::TextReader::Close, System::IO::TextWriter::Close, System::IO::StringReader::Close, System::IO::StringWriter::Close, System::IO::StreamReader::Close, System::IO::StreamWriter::Close | |
Associate stream with file descriptor of open file | System::IO::FileStream::FileStream | |
Test for end of file on stream | ||
Test for error on stream | Not applicable. To call the standard C function, use PInvoke. For more information, see Platform Invoke Examples. | |
Flush stream to buffer or storage device | ||
Read character from stream (function versions of getc and getwc) | ||
Read character from stdin (function versions of getchar and getwchar) | ||
Get position indicator of stream | ||
Read string from stream | System::IO::StreamReader::ReadLine, System::IO::TextReader::ReadBlock | |
Get file descriptor associated with stream | ||
Flush all streams to buffer or storage device | System::IO::FileStream::Flush, System::IO::StreamWriter::Flush, System::IO::TextWriter::Flush, System::IO::BinaryWriter::Flush | |
Open stream | ||
fprintf, _fprintf_l, fwprintf, _fwprintf_l, fprintf_s, _fprintf_s_l, fwprintf_s, _fwprintf_s_l | Write formatted data to stream | |
Write a character to a stream (function versions of putc and putwc) | ||
Write character to stdout (function versions of putchar and putwchar) | ||
Write string to stream | ||
Read unformatted data from stream | ||
Reassign FILE stream pointer to new file or device | ||
fscanf, fwscanf, fscanf_s, _fscanf_s_l, fwscanf_s, _fwscanf_s_l | Read formatted data from stream | System::IO::StreamReader::ReadLine; see also Parse methods, such as System::Double::Parse. |
Move file position to given location | System::IO::FileStream::Position, System::IO::FileStream::Seek | |
Set position indicator of stream | ||
Open stream with file sharing | Not applicable. To call the standard C function, use PInvoke. For more information, see Platform Invoke Examples. | |
Get current file position | ||
Write unformatted data items to stream | ||
Read character from stream (macro versions of fgetc and fgetwc) | ||
Read character from stdin (macro versions of fgetchar and fgetwchar) | ||
Returns the number of simultaneously open files permitted at the stream I/O level. | Not applicable. To call the standard C function, use PInvoke. For more information, see Platform Invoke Examples. | |
Read line from stdin | ||
Read binary int from stream | Not applicable. To call the standard C function, use PInvoke. For more information, see Platform Invoke Examples. | |
printf, _printf_l, wprintf, _wprintf_l, printf_s, _printf_s_l, wprintf_s, _wprintf_s_l | Write formatted data to stdout | |
Write character to a stream (macro versions of fputc and fputwc) | ||
Write character to stdout (macro versions of fputchar and fputwchar) | ||
Write line to stream | ||
Write binary int to stream | Not applicable. To call the standard C function, use PInvoke. For more information, see Platform Invoke Examples. | |
Move file position to beginning of stream | Not applicable. To call the standard C function, use PInvoke. For more information, see Platform Invoke Examples. | |
Remove temporary files created by tmpfile | Not applicable. To call the standard C function, use PInvoke. For more information, see Platform Invoke Examples. | |
scanf, _scanf_l, wscanf, _wscanf_l, scanf_s, _scanf_s_l, wscanf_s, _wscanf_s_l | Read formatted data from stdin | System::Console::ReadLine; see also Parse methods, such as System::Double::Parse. |
Control stream buffering | Not applicable. To call the standard C function, use PInvoke. For more information, see Platform Invoke Examples. | |
Set a maximum for the number of simultaneously open files at the stream I/O level. | Not applicable. To call the standard C function, use PInvoke. For more information, see Platform Invoke Examples. | |
Control stream buffering and buffer size | Not applicable. To call the standard C function, use PInvoke. For more information, see Platform Invoke Examples. | |
_snprintf, _snwprintf, _snprintf_s, _snprintf_s_l, _snwprintf_s, _snwprintf_s_l | Write formatted data of specified length to string | Not applicable. To call the standard C function, use PInvoke. For more information, see Platform Invoke Examples. |
_snscanf, _snwscanf, _snscanf_s, _snscanf_s_l, _snwscanf_s, _snwscanf_s_l | Read formatted data of a specified length from the standard input stream. | Not applicable. To call the standard C function, use PInvoke. For more information, see Platform Invoke Examples. |
sprintf, swprintf, sprintf_s, _sprintf_s_l, swprintf_s, _swprintf_s_l | Write formatted data to string | |
sscanf, swscanf, sscanf_s, _sscanf_s_l, swscanf_s, _swscanf_s_l | Read formatted data from string | See Parse methods, such as System::Double::Parse |
Generate temporary filename in given directory | Not applicable. To call the standard C function, use PInvoke. For more information, see Platform Invoke Examples. | |
Create temporary file | Not applicable. To call the standard C function, use PInvoke. For more information, see Platform Invoke Examples. | |
Generate temporary filename | Not applicable. To call the standard C function, use PInvoke. For more information, see Platform Invoke Examples. | |
Push character back onto stream | Not applicable. To call the standard C function, use PInvoke. For more information, see Platform Invoke Examples. | |
_vcprintf, _vcwprintf, _vcprintf_s, _vcprintf_s_l, _vcwprintf_s, _vcwprintf_s_l | Write formatted data to the console. | |
vfprintf, vfwprintf, vfprintf_s, _vfprintf_s_l, vfwprintf_s, _vfwprintf_s_l | Write formatted data to stream | Not applicable. To call the standard C function, use PInvoke. For more information, see Platform Invoke Examples. |
vprintf, vwprintf, vprintf_s, _vprintf_s_l, vwprintf_s, _vwprintf_s_l | Write formatted data to stdout | |
_vsnprintf, _vsnwprintf, vsnprintf_s, _vsnprintf_s, _vsnprintf_s_l, _vsnwprintf_s, _vsnwprintf_s_l | Write formatted data of specified length to buffer | Not applicable. To call the standard C function, use PInvoke. For more information, see Platform Invoke Examples. |
vsprintf, vswprintf, vsprintf_s, _vsprintf_s_l, vswprintf_s, _vswprintf_s_l | Write formatted data to buffer |
Low-Level I/O
These functions invoke the operating system directly for lower-level operation than that provided by stream I/O. Low-level input and output calls do not buffer or format data.
Low-level routines can access the standard streams opened at program startup using the following predefined file descriptors.
Function | Use |
---|---|
Close file | |
Flush file to disk | |
Create file | |
Return next available file descriptor for given file | |
Create second descriptor for given file | |
Test for end of file | |
Reposition file pointer to given location | |
Open file | |
Read data from file | |
Open file for file sharing | |
Get current file-pointer position | |
Set file-permission mask | |
Write data to file |