TooManyOpenFiles

http://wiki.apache.org/hadoop/CouldOnlyBeReplicatedTo

Too Many Open Files

You can see this on Linux machines in client-side applications, server code or even in test runs.

It is caused by per-process limits on the number of files that a single user/process can have open, which was introduced inthe 2.6.27 kernel. The default value, 128, was chosen because "that should be enough".

In Hadoop, it isn't. To fix this log in/su/ssh as root and edit/etc/sysctl.conf

add the line

fs.epoll.max_user_instances = 20480

Then reboot. Different numbers may be chosen.

There is an immediate shortcut: echo 2048 > /proc/sys/fs/epoll/max_user_instances . This setting will be lost on the next reboot, but is handy for trying out different values.

limits.conf

Another limit on the number of files open may be the file /etc/security/limits.conf

It has a setting on the number of files a user or group may have,nofile.

To set this, as root edit /etc/security/limits.conf

and add a line such as

*               soft    nofile            20480
*               hard    nofile            20480

Then restart terminal to take effect.

To see the current/default limits, run the command ulimit -a. This should print something like

core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 20
file size               (blocks, -f) unlimited
pending signals                 (-i) 16382
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 4096
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) unlimited
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

You can dynamically up the limits until the next reboot with the same command. Specifically

ulimit -n 8192

The updated value can then be printed

# ulimit -n
8192