在 UNIX/Linux 系统中,一切皆文件,这句话想必都有听过。对于文件的操作几乎适用于所有的设备,这也就看出了文件操作的重要性了。在C语言再学习部分有讲过标准I/O文件操作,参看:C语言再学习 -- 文件 下面我们来讲解下系统文件I/O的。
一、文件描述符
1、文件描述符简介
首先从文件描述符开始讲起。因为,对于内核而言,所有打开的文件都是通过文件描述符引用的。那么文件描述符到底是什么?
文件描述符(file descriptor)通常是一个小的非负整数,内核用以标识一个特定进程正在访问的文件。当打开一个现有文件或创建一个新文件时,内核向进程返回一个文件描述符。当读、写一个文件时,使用 open 或 create 返回的文件描述符标识该文件,将其作为参数传送给 read 或 write。
2、标准输入、标准输出和标准错误
按照惯例,UNIX 系统 shell 把文件描述符 0 与进程的标准输入(standard input)关联,文件描述符 1 与标准输出(standard output)关联,文件描述符 2 与标准错误(standard error)关联。这是各种 shell 以及很多应用程序使用的惯例,与 UNIX 内核无关。尽管如此,如果不遵循这种惯例,很多 UNIX 系统应用程序就不能正常工作。
这部分讲 shell 编程重定向时,可参看UNIX再学习 -- shell编程
POSIX 定义了 STDIN_FILENO、STDOUT_FILENO 和 STDERR_FILENO 来代替 0、1、2。这三个符号常量的定义位于头文件 <unistd.h>
-
查看 /usr/include/unistd.h
-
/* Standard file descriptors. */
-
#define STDIN_FILENO 0 /* Standard input. */
-
#define STDOUT_FILENO 1 /* Standard output. */
-
#define STDERR_FILENO 2 /* Standard error output. */
文件描述符的有效范围是 0 到 OPEN_MAX。一般来说,每个进程最多可以打开 64 个文件(0 — 63)。对于 FreeBSD 8.0、Linux 3.2.0、Mac OS X 10.6.8 以及 Solaris 10 来说,文件描述符的变化范围几乎是无限的,它只受系统配置的存储器总量、整型的字长以及系统管理员所配置的软限制和硬限制的约束。OPEN_MAX:每个进程最大打开文件数。查看如下:
-
查看 man sysconf
-
PEN_MAX - _SC_OPEN_MAX
-
The maximum number of files that a process can have open at any time. Must
not be less than _POSIX_OPEN_MAX (
20).
或参看:man sysconf()
-
#include <unistd.h>
-
#include <stdio.h>
-
int main (void)
-
{
-
printf(
"_SC_ARG_MAX: %ld\n", sysconf(_SC_ARG_MAX));
-
printf(
"_SC_CHILD_MAX: %ld\n", sysconf(_SC_CHILD_MAX));
-
printf(
"_SC_CLK_TCK: %ld\n", sysconf(_SC_CLK_TCK));
-
printf(
"_SC_NGROUPS_MAX: %ld\n", sysconf(_SC_NGROUPS_MAX));
-
printf(
"_SC_OPEN_MAX: %ld\n", sysconf(_SC_OPEN_MAX));
-
printf(
"_SC_JOB_CONTROL: %ld\n", sysconf(_SC_JOB_CONTROL));
-
printf(
"_SC_SAVED_IDS: %ld\n", sysconf(_SC_SAVED_IDS));
-
printf(
"_SC_VERSION: %ld\n", sysconf(_SC_VERSION));
-
return
0;
-
}
-
输出结果:
-
_SC_ARG_MAX:
2097152
-
_SC_CHILD_MAX:
7892
-
_SC_CLK_TCK:
100
-
_SC_NGROUPS_MAX:
65536
-
_SC_OPEN_MAX:
1024
-
_SC_JOB_CONTROL:
1
-
_SC_SAVED_IDS:
1
-
_SC_VERSION:
200809
可得,在 Ubuntu 12.04 下测试结果为:_SC_OPEN_MAX =1024
而在 /apue.3e/exercises/openmax.c 也是有获取系统的 _SC_OPEN_MAX 的程序,可自行查看。
3、最大文件描述符数
通过上面可知,默认的 Linux 进程的最大打开文件描述符数是 1024。但是在某些情况下这点文件描述符数是远远不够的,可根据需要进行更改。
第一种方法:通过 ulimit 命令修改
-
选项 [options] 含义 例子
-
-H 设置硬资源限制,一旦设置不能增加。 ulimit – Hs
64;限制硬资源,线程栈大小为
64K。
-
-S 设置软资源限制,设置后可以增加,但是不能超过硬资源设置。 ulimit – Sn
32;限制软资源,
32 个文件描述符。
-
-a 显示当前所有的 limit 信息。 ulimit – a;显示当前所有的 limit 信息。
-
-c 最大的 core 文件的大小, 以 blocks 为单位。 ulimit – c unlimited; 对生成的 core 文件的大小不进行限制。
-
-d 进程最大的数据段的大小,以 Kbytes 为单位。 ulimit -d unlimited;对进程的数据段大小不进行限制。
-
-f 进程可以创建文件的最大值,以 blocks 为单位。 ulimit – f
2048;限制进程可以创建的最大文件大小为
2048 blocks。
-
-l 最大可加锁内存大小,以 Kbytes 为单位。 ulimit – l
32;限制最大可加锁内存大小为
32 Kbytes。
-
-m 最大内存大小,以 Kbytes 为单位。 ulimit – m unlimited;对最大内存不进行限制。
-
-n 可以打开最大文件描述符的数量。 ulimit – n
128;限制最大可以使用
128 个文件描述符。
-
-p 管道缓冲区的大小,以 Kbytes 为单位。 ulimit – p
512;限制管道缓冲区的大小为
512 Kbytes。
-
-s 线程栈大小,以 Kbytes 为单位。 ulimit – s
512;限制线程栈的大小为
512 Kbytes。
-
-t 最大的 CPU 占用时间,以秒为单位。 ulimit – t unlimited;对最大的 CPU 占用时间不进行限制。
-
-u 用户最大可用的进程数。 ulimit – u
64;限制用户最多可以使用
64 个进程。
-
-v 进程最大可用的虚拟内存,以 Kbytes 为单位。 ulimit – v
200000;限制最大可用的虚拟内存为
200000 Kbytes。
-
ulimit [-HSTabcdefilmnpqrstuvx [limit]]
-
Provides control over the resources available to the shell
and to processes started by it, on systems that allow such con‐
-
trol. The -H
and -S options specify that the hard
or soft limit is
set
for the given resource. A hard limit cannot be
-
increased by a non-root user once it is
set; a soft limit may be increased up to the value of the hard limit. If neither -H
-
nor -S is specified, both the soft
and hard limits are
set. The value of limit can be a number in the unit specified
for the
-
resource
or one of the special values hard, soft,
or unlimited, which stand
for the current hard limit, the current soft
-
limit,
and no limit, respectively. If limit is omitted, the current value of the soft limit of the resource is printed,
-
unless the -H option is given. When more than one resource is specified, the limit name
and unit are printed before the
-
value. Other options are interpreted as follows:
-
-a All current limits are reported
-
-b The maximum socket buffer size
-
-c The maximum size of core files created
-
-d The maximum size of a process's data segment
-
-
e The maximum scheduling priority ("nice")
-
-f The maximum size of files written by the shell and its children
-
-i The maximum number of pending signals
-
-l The maximum size that may be locked into memory
-
-m The maximum resident set size (many systems do not honor this limit)
-
-n The maximum number of open file descriptors (most systems do not allow this value to be set)
-
-p The pipe size in 512-byte blocks (this may not be set)
-
-q The maximum number of bytes in POSIX message queues
-
-r The maximum real-time scheduling priority
-
-s The maximum stack size
-
-t The maximum amount of cpu time in seconds
-
-u The maximum number of processes available to a single user
-
-v The maximum amount of virtual memory available to the shell and, on some systems, to its children
-
-x The maximum number of file locks
-
-T The maximum number of threads
-
-
If limit is given, it is the new value of the specified resource (the -a option is display only). If no option is given,
-
then -f is assumed. Values are in 1024-byte increments, except for -t, which is in seconds, -p, which is in units of
-
512-byte blocks, and -T, -b, -n, and -u, which are unscaled values. The return status is 0 unless an invalid option or argu‐
-
ment is supplied, or an error occurs while setting a new limit.
显示当前最大打开文件描述符数
-
# ulimit -n
-
1024
修改当前用户环境下的最大打开文件描述符数 (临时更改)
-
设置当前用户环境下的最大打开文件描述符数为
65536
-
# ulimit -HSn 65536
-
-
查看:
-
# ulimit -n
-
65536
将指令添加到脚本中 (永久更改)
添加到 单用户目录下:/etc/bash.bashrc 或 ~/.bashrc
-
echo
"ulimit -HSn 65536" >> ~/.bashrc
-
或者
-
echo
"ulimit -HSn 65536" >> /etc/bash.bashrc
我使用的是 root 超级用户登录,而非 non-root 登录的,所以放在 /etc/profile 等针对所有用户的不起作用。
还有网上说的写到 rc.local 我也没有实现。
第二种方法:修改 limits.conf 文件 (永久更改)
在 /etc/security/limits.conf 文件最后加入如下两行:
-
* soft nofile
65536
-
* hard nofile
65536
其中 * 代表所有用户,nofile 是代表最大文件打开数。用 non-root 登录,通过 ulimit -n 查看是否生效。
告诉你个不幸的消息,很遗憾我的没有生效。将 * 改为 root 则可以生效,因为我用的不是 non-root 登录的。
查看最大文件描述符数上限
-
# ulimit -n 100000000
-
bash: ulimit: open files: 无法修改 limit 值: 不允许的操作
而最大文件描述符数的上限值是在 /proc/sys/fs/nr_open 设置的,默认为 1048576
echo 2000000 > /proc/sys/fs/nr_open
4、混淆的概念
-
file-max & file-nr:
-
-
The kernel allocates file handles dynamically, but as yet it
-
doesn't
free them again.
-
-
The value in file-max denotes the maximum number of file-
-
handles that the Linux kernel will allocate. When you get lots
-
of error messages about running out of file handles, you might
-
want to increase
this limit.
-
-
Historically, the three values in file-nr denoted the number of
-
allocated file handles, the number of allocated but unused file
-
handles,
and the maximum number of file handles. Linux
2.6 always
-
reports
0 as the number of
free file handles --
this is
not an
-
error, it just means that the number of allocated file handles
-
exactly matches the number of used file handles.
-
-
Attempts to allocate more file descriptors than file-max are
-
reported with printk, look
for
"VFS: file-max limit <number>
-
reached".
-
nr_open:
-
-
This denotes the maximum number of file-handles a process can
-
allocate. Default value is
1024*
1024 (
1048576) which should be
-
enough
for most machines. Actual limit depends on RLIMIT_NOFILE
-
resource limit.
翻译一下:
(1)概念解析
先说 file_max
-
/proc/sys/fs/file-max
-
This file defines a system-wide limit on the number of open files
for all processes. (See also setrlimit(
2), which can be
-
used by a process to
set the per-process limit, RLIMIT_NOFILE, on the number of files it may open.) If you get lots of error
-
messages about running out of file handles,
try increasing
this value:
-
-
echo
100000 > /proc/sys/fs/file-max
-
-
The kernel constant NR_OPEN imposes an upper limit on the value that may be placed in file-max.
-
-
If you increase /proc/sys/fs/file-max, be sure to increase /proc/sys/fs/inode-max to
3
-4 times the
new value of
-
/proc/sys/fs/file-max,
or you will run out of inodes.
翻译一下:
-
# cat /proc/sys/fs/file-max
-
100987
-
# grep MemTotal /proc/meminfo | awk '{printf("%d\n",$2/10)}'
-
102479
经比较,可能由于有各种其他原因导致 file-max 没有设置为内存的 10%。
-
# cat /proc/sys/fs/file-nr
-
6464
0
100987
echo 1000000 > /proc/sys/fs/file-max
第二种:永久更改
(2)系统级 和 进程级 的区别
首先,系统级
-
# cat /proc/sys/fs/file-nr
-
6464
0
100987
file-nr 中的值由三部分组成,分别为:1.已经分配的文件句柄数,2.已经分配但没有使用的文件句柄数,3.最大文件句柄数。也就是说,系统文件描述符的数量目前被使用 6464.
![](https://i-blog.csdnimg.cn/blog_migrate/b51985c17e6dace42847c65d3fde5eec.png)
使用 ps -aux | more 表示将ps -aux的结果进行分屏显示
-
# ps
-
PID TTY TIME CMD
-
2401 pts/
1
00:
00:
00 bash
-
3054 pts/
1
00:
00:
00 ps
-
# ps -aux | more
-
Warning: bad ps syntax, perhaps a bogus
'-'? See http:
//procps.sf.net/faq.html
-
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
-
root
1
0.0
0.1
3612
2024 ? Ss
10:
53
0:
01 /sbin/init
-
root
2
0.0
0.0
0
0 ? S
10:
53
0:
00 [kthreadd]
-
root
3
0.0
0.0
0
0 ? S
10:
53
0:
00 [ksoftirqd/
0]
-
root
6
0.0
0.0
0
0 ? S
10:
53
0:
00 [migration/
0]
-
root
7
0.0
0.0
0
0 ? S
10:
53
0:
01 [watchdog/
0]
-
root
8
0.0
0.0
0
0 ? S<
10:
53
0:
00 [cpuset]
-
root
9
0.0
0.0
0
0 ? S<
10:
53
0:
00 [khelper]
-
root
10
0.0
0.0
0
0 ? S
10:
53
0:
00 [kdevtmpfs]
-
.....
那我们就看看当前终端启动的进程 2401
-
# lsof | grep '2401' | wc -l
-
20
-
-
意思是有
20个文件被打开
-
# ls -l /proc/2401/fd/ | wc -l
-
5
-
-
意思是文件描述符只有
5个
所以说,一个文件即使被打开,也可能没有文件描述符,比如当前工作目录,内存映射文件和可执行文本文件。再者,进程级
-
//open函数和close函数的使用
-
#include <stdio.h>
-
#include <stdlib.h>
-
#include <unistd.h>
-
#include <sys/types.h>
-
#include <sys/stat.h>
-
#include <fcntl.h>
-
-
int main(void)
-
{
-
-
// while (1){
-
int fd = open(
"a.txt",O_WRONLY|O_CREAT|O_TRUNC,
0644);
-
if(
-1 == fd)
-
{
-
perror(
"open"),
exit(
-1);
-
}
-
-
-
printf (
"fd = %d\n", fd);
-
#if 1
-
//2.关闭文件
-
int res = close(fd);
-
if(
-1 == res)
-
{
-
perror(
"close"),
exit(
-1);
-
}
-
-
#endif
-
// }
-
return
0;
-
}
-
输出结果:
-
fd =
3
该程序,除去标准输入(0)、标准输出(1)、标准错误(2),文件描述符从 3 开始。
-
忽略....
-
fd =
1019
-
fd =
1020
-
fd =
1021
-
fd =
1022
-
fd =
1023
-
open: Too many open files
-
# ulimit -n
-
1024
5、文件描述符和文件指针
(1)文件描述符和文件指针比较
-
struct _IO_FILE {
-
int _flags;
/* High-order word is _IO_MAGIC; rest is flags. */
-
#define _IO_file_flags _flags
-
-
/* The following pointers correspond to the C++ streambuf protocol. */
-
/* Note: Tk uses the _IO_read_ptr and _IO_read_end fields directly. */
-
char* _IO_read_ptr;
/* Current read pointer */
-
char* _IO_read_end;
/* End of get area. */
-
char* _IO_read_base;
/* Start of putback+get area. */
-
char* _IO_write_base;
/* Start of put area. */
-
char* _IO_write_ptr;
/* Current put pointer. */
-
char* _IO_write_end;
/* End of put area. */
-
char* _IO_buf_base;
/* Start of reserve area. */
-
char* _IO_buf_end;
/* End of reserve area. */
-
/* The following fields are used to support backing up and undo. */
-
char *_IO_save_base;
/* Pointer to start of non-current get area. */
-
char *_IO_backup_base;
/* Pointer to first valid character of backup area */
-
char *_IO_save_end;
/* Pointer to end of non-current get area. */
-
-
struct _IO_marker *_markers;
-
-
struct _IO_FILE *_chain;
-
-
int _fileno;
-
#if 0
-
int _blksize;
-
#else
-
int _flags2;
-
#endif
-
_IO_off_t _old_offset;
/* This used to be _offset but it's too small. */
-
-
#define __HAVE_COLUMN /* temporary */
-
/* 1+column number of pbase(); 0 is unknown. */
-
unsigned
short _cur_column;
-
signed
char _vtable_offset;
-
char _shortbuf[
1];
-
-
/* char* _save_gptr; char* _save_egptr; */
-
-
_IO_lock_t *_lock;
-
#ifdef _IO_USE_OLD_IO_FILE
-
};
-
/* Define outside of namespace so the C++ is happy. */
-
struct _IO_FILE;
-
-
__BEGIN_NAMESPACE_STD
-
/* The opaque type of streams. This is the definition used elsewhere. */
-
typedef
struct _IO_FILE FILE;
-
__END_NAMESPACE_STD
-
#if defined __USE_LARGEFILE64 || defined __USE_SVID || defined __USE_POSIX \
-
|| defined __USE_BSD || defined __USE_ISOC99 || defined __USE_XOPEN \
-
|| defined __USE_POSIX2
-
__USING_NAMESPACE_STD(FILE)
-
#endif
-
-
# define __FILE_defined 1
-
#endif /* FILE not defined. */
-
#undef __need_FILE
-
-
-
#if !defined ____FILE_defined && defined __need___FILE
-
-
/* The opaque type of streams. This is the definition used elsewhere. */
-
typedef
struct _IO_FILE __FILE;
-
#include<stdio.h>
-
#include<stdlib.h>
-
#include<sys/types.h>
-
#include<sys/stat.h>
-
#include<fcntl.h>
-
#include<unistd.h>
-
-
int main(){
-
char buf[
50] = {
"file descriptor demo"};
-
FILE *myfile;
-
-
myfile = fopen(
"test",
"w+");
-
if(!myfile){
-
printf(
"error: openfile failed!\n");
-
}
-
printf(
"The openfile's descriptor is %d\n", myfile->_fileno);
-
if(write(myfile->_fileno,buf,
50)<
0){
-
perror(
"error: write file failed!\n");
-
exit(
1);
-
}
else{
-
printf(
"writefile successed!\n");
-
}
-
-
exit(
0);
-
}
-
输出结果:
-
The openfile's descriptor is
3
-
writefile successed!
-
-
查看 test
-
cat test
-
file descriptor demoroot
(2)文件描述符和文件指针相互转换
文件指针转文件描述符:
-
#include <stdio.h>
-
int main( void )
-
{
-
printf(
"The file descriptor for stdin is %d\n", fileno(
stdin ) );
-
printf(
"The file descriptor for stdout is %d\n", fileno(
stdout ) );
-
printf(
"The file descriptor for stderr is %d\n", fileno(
stderr ) );
-
return
0;
-
}
-
输出结果:
-
The file descriptor
for
stdin is
0
-
The file descriptor
for
stdout is
1
-
The file descriptor
for
stderr is
2
-
#include <stdio.h>
-
int main(void)
-
{
-
FILE *fp;
-
int fd;
-
fp = fopen(
"/etc/passwd",
"r");
-
fd = fileno(fp);
-
printf(
"fd = %d\n", fd);
-
fclose(fp);
-
return
0;
-
}
-
输出结果:
-
fd =
3
文件描述符转文件指针:
FILE* fdopen(int fd, const char* type);
fdopen 取一个现存的文件描述符,并使一个标准的I / O流与该描述符相结合。此函数常用于由创建管道和网络通信通道函数获得的描述符。因为这些特殊类型的文件不能用标准I/O fopen函数打开,首先必须先调用设备专用函数以获得一个文件描述符,然后用fdopen使一个标准I/O流与该描述符相结合。
-
#include<stdio.h>
-
int main (void)
-
{
-
FILE * fp = fdopen (
1,
"w+");
-
fprintf (fp,
"%s\n",
"hello!");
-
fclose (fp);
-
return
0;
-
}
-
输出结果:
-
hello!